parallel miniscope recordings: disconnections not apearing in Control Panel

[arnauramos1]

Hi all,
I’ve been experiencing a weird bug that could not find an explanation for anywhere in the forums:

I have a setup with 2-4 miniscopes running in parallel. During recordings, every now and then, one of the miniscopes will “restart” and go focal plane 0, 30fps (I image at 20fps and different focal plane; specified within the .json file), and the gyroscope stop working.
This error does not pop up in the control panel, and the three lights in the daq still appear green.
The only solution I found to save the data during recordings is to disconnect and reconnect the miniscope, and then manually adjust the focus and fps.
I have tried several things without success.. new cables, new daqs, disconnecting the pc from the network… My PC is quite potent, and the simultaneous recordings do not even make the cpu, gpu or ram go to 50%.

Any idea why this could be happening? Any insight would be highly appreciated!

Cheers,
Arnau

[sneakers-the-rat]

I diagnosed a problem that would show up like this a few months ago, but didn't open it since we are actively in the process of replacing this software (bear with us for the next few months as we make a swap :)

i'll repost it here in the morning, but for now:

• is there nothing printed in any of your console or logfiles at all? or if you ctrl+f can you find any that say "Frames will be lost" ?
• is it always the same camera, or is it random?
• is it always after ~some duration or can it happen near the start of the recording?
• are you saving to a spinning disk hard drive or an SSD?

[arnauramos1]

Hi,
Thanks for the prompt answer.

• I don't have anything printed.
• It is random.
• It can happen at any timepoint.
• I am saving to an SSD.

Thanks for the help!

[arnauramos1]

If instead of having 4 miniscopes at 20fps I have them at 30fps at the start (i.e. already specified at the json file), then I get this notification of "frames will be lost", even though the frame buffer does not appear full, and that notification always appears for the same miniscope (my device2).

[sneakers-the-rat]

Aha yeah, so there's youre getting to the problem that i thought it was, where the way the data saving loop is written has it loop through each camera, draining its queue before going on to the next camera:

Miniscope-DAQ-QT-Software/source/datasaver.cpp

Lines 184 to 186 in 0fecb97

	for (i = 0; i < frameBuffer.size(); i++) {
	while (usedCount[names[i]]->tryAcquire()) {
	// grad info from buffer in a threadsafe way

and the size of the memory buffer is pretty small (128 frames) -

Miniscope-DAQ-QT-Software/source/videodevice.h

Line 43 in 0fecb97

#define FRAME_BUFFER_SIZE 128

so that "frames will be lost" message starts happening when we are write limited.

If you're writing to an SSD and not really i/o limited, if you're using any kind of compression you could disable that and then compress afterwards, see if that helps the frame dropping problem - even if you aren't maxing out your CPU, that current DAQ doesn't have great concurrency so it could be at the limit of single-threaded video encoding.

For the original problem, i don't really have any good idea, and as far as i'm aware this software doesn't have any good way of printing more detailed debugging information, but the fact that it loses its configuration is odd. i'm don't have a clear picture of how the software would handle recognizing (and thus re-using the config) a device if it e.g. had a hardware problem and restarted/lost connection/etc., but if it can be any of your cameras at any time and you're not being bound by CPU that's mysterious - presumably not every one of your miniscopes has a hardware bug. maybe @daharoni or @fnsangiul have a better idea.

[fnsangiul]

Based on my experience recording with multiple imaging devices (Miniscopes and miniCAMs) the disconnection happens when the animal hits the Miniscope against something, disconnecting the Miniscope from the computer. Sometimes it reconnects automatically but when it doesn't link again with the computer there is no other way than manually disconnecting and connecting the coaxial cable to the DAQ. When it reconnects, it goes into the default settings.

[sneakers-the-rat]

aha. ok we'll test for that in mio (which fyi @arnauramos1 will replace this software), we'll figure out a way to poll & reset on disconnection and persist config

[fnsangiul]

@sneakers-the-rat I don't know if it's possible to do it via software because the Miniscope loses the link to the serializer. That's why you need to reset the DAQ, either by connecting the coax cable again or resetting it by unplugging the DAQ.

[sneakers-the-rat]

you can't trigger the DAQ to reset the serializer? how does device discovery work when starting from scratch with a config vs. reconnecting after connection loss that makes it impossible to preserve the config? seems like it should be possible to poll from software to the daq like "keep resetting until something reconnects"

[fnsangiul]

I'm referring to one step back. If the Miniscope loses the data link then I don't think you can reset it via software. You can check the deserializer datasheet for this though.

[sneakers-the-rat]

i guess another way of putting it is why does disconnecting and reconnecting the cable fix it, and why can't we trigger that same reset from code?

[fnsangiul]

I wouldn't say it fixes it, what happens is that the Miniscope locks again. Here is the datasheet for the deserializer.

[sneakers-the-rat]

for the sake of this issue - what can we do here? is there something we can improve on/fix so that the user experience doesn't include having to unplug and replug-in the cable to fix problems that we should happen to expect? (mice knocking the cameras around). I am not sure what you're showing me here with the datasheet, i'm not sure how it relates to the issue?

[fnsangiul]

@sneakers-the-rat I don't think there is another way other than unplugging and replugin the coax cable, or at least I haven't found one. As for the datasheet I linked it because if I had to isolate this particular issue it would be due to the deserializer.

[sneakers-the-rat]

So there is no way that we could, e.g. in the firmware of the DAQ, to do whatever state reset happens when one physically unplugs/replugs the cable? like what does unplugging/plugging it back in do to get the miniscope recognized again, and can we automate that so OP doesn't need to babysit the cameras and plug/unplug cables?

Aside from that, it should be the case that the software alerts you when a camera disconnects in some way, so even if there is no way to trigger a virtual unplug/replug via the software or firmware we can at least let the user know that they need to do that. but unfortunately this software is deprecated and we are in the process of replacing it, so we will make sure that there are informative error messages for this problem. doesn't help much on this issue for now, but that's the state of development we're at.

one more question for @arnauramos1 - does this ever happen when just one camera is connected? like if the source of the problem is just connectivity issues when the mouse is bumping around, then we would expect that to also happen when there is just one camera, but if it only happens when there are multiple cameras that seems to suggest there is something else going on too

[arnauramos1]

Hey,
Thanks very much both of you for taking so much care of this!
I haven't tested it with one camera only for this specific test since it is quite long (1h approx). In other similar tests where I have tested only 2 cameras +2 miniscopes, these disconnections did not happen.
I can already say that the disconnection is not "motion/bumping related", since I have checked the moments when it disconnected and the mouse is sometimes just freezing..
I also checked that the disconnection wouldn't be "power" related, by having superspeed usb hubs with external power connected, as well as the daq powered externally too. Still happened.

[sneakers-the-rat]

Thanks for the troubleshooting :) if it isn't motion/bumping related, we have a bunch of testing to do - again i'll say this software in particular is deprecated and is actively being worked on to be replaced, which again is not satisfying! but we are implementing something that will be a) much more reliable by design, and part of that is b) making it possible to understand why problems happen when they happen.

imo it's on us atp to understand this problem since we don't have great logging visiblity in the hardware, firmware, and software - let's rig up a multicam setup and try and reproduce the bug.

[sneakers-the-rat]

one last thing - for reproduciblity's sake @arnauramos1 could you post the config file you are running with? we also don't have issue templates set up on this repo to prompt for them (since this software is deprecated)

[arnauramos1]

round1.json

Hey, sorry for my late response.. here it is.

[sneakers-the-rat]

Hmm ya that looks normal, i mostly wanted to see compression settings because that's the major software bottleneck i've seen happen, you might try flipping that to GRAY but i don't have a great deal of confidence that's even related.

I'm sorry we don't have a super satisfying answer for you here at the moment, Federico is slammed for time, but said he would bet that it was the uFL connector on the miniscope and not a software problem. he says:

The solution is to remove the connector on uFL connector on the miniscope and the coax cable and solder the cable directly on the pads. That's way more stable

He knows the system about a million times better than I do so that might be a starting point, although i don't know exactly what you would do to diagnose whether that is the problem and can't guide you much there, and i'm not sure how reversible that might be.

I still think there is some possibility that this is a software or firmware bug since it happens at random times, on random scopes, doesn't happen when you use fewer scopes, isn't caused by movement or bumping, but since the bug you're experiencing is exactly that the software doesn't give you any notice and it doesn't produce any debug logs beyond that there also isn't much i can do to help diagnose that either. I am actively working on replacing this software for these and other reasons, and until that happens we're unfortunately in a position where we can't fix things.

Another option you might try just to rule out software bugs before trying the soldering route would be to try using Bonsai for capture instead of this software. To avoid wasting time with your animals you could just try running that e.g. overnight with the cameras not mounted.

Sorry again we are in a bit of a limbo on our ability to handle bugs, working on it! @daharoni may have more information but is also PI levels of busy