Occasionally, a ZED 2i will completely drop out while running. This is characterized by:
Completely black RGB image frames
Randomly incomplete and misaligned depth/normal image frames and depth point clouds
Apparent loss of horizontal sync visible in confidence image frames
The only way to recover the camera from this state is to reboot it (closing + opening it with the SDK does not recover the camera).
Other things I’ve noticed:
Observed on both Windows 10 and Ubuntu 22 (i.e. seems to be a hardware/firmware issue)
Observed on SDK 3.8.2 and 4.0.7
Observed on camera firmware versions 1523
Observed on many different USB setups (varying cable lengths and powered hub / injector setups); does not seem to be a USB issue
Observed on different cameras (i.e. not a defect in a specific camera)
Very rare: In the past month I’ve only seen it happen twice, on three cameras. Also it has reportedly happened a handful of times over the past year on another camera that we own.
I have captured video (sorry for length) of it happening at https://youtu.be/Y4x-yuedEmY (see chapter markers). In the confidence images, you can clearly see some sort of horizontal sync issue. (Note: The image on the right in the video is from a second camera, it is NOT the right sensor image.)
The workaround we currently have implemented is to check the RGB images for completely black frames and reboot the camera if too many are seen in a row. This presents a number of performance problems.
It would be nice to have Stereolabs take a look at this, as it’s been presenting a consistent reliability problem for us.
Unfortunately, I just realized that I neglected to check if the right sensor was also showing issues, or if it was unique to the left . If/when it happens again and I am able to collect that data, I will update here.
We introduced a new InitParameter some versions ago, enable_image_validity_check. This allows getting an error message when the frames are detected invalid, and it would cover the black frame detection well. You could call the reboot method when an invalid frame is detected. It may help getting some performance back.
If the connection is lost sometimes, you might also want to enable async_grab_camera_recovery. It can help if the camera is up for long periods of time.
It seems SDK 4.0.8 is required for that (ERROR_CODE.CORRUPT_FRAME is not defined in the Python API in 4.0.7, only in 4.0.8). Unfortunately, SDK 4.0.8 is unusable because of this issue, so I can’t take advantage of it.
But once the logging issue is fixed I will use the built in corrupt frame detection.
Still, it would be ideal if the camera firmware was updated to fix the actual underlying issue.
Yeah I got it, thanks. I switched over to using the built-in corrupt frame detection. Haven’t had a camera drop out yet but next time it happens I’ll get to see if it works.
It may be hard to see but the image is only almost black, with a few black vertical stripes in it.
So until the hsync (presumably) issue is corrected, we need another more reliable way to detect when this is happening so we can automate camera reboots.
Detection methods that definitely fail are:
Looking for all black images: the images are not all black
Looking for frames with no depth cloud points: there are still some depth cloud points in these frames
All I can think of is to look and see if the image is very dark, but this runs the risk of having false positive detections if something is obstructing the camera or if the lights are off in the space the camera is in.
Any ideas would be appreciated; or at least some reassurance that the underlying issue is being worked on.
@JC3 I’m discussing this with the team about this, I’ll provide you with any leads we get.
The most efficient way to investigate this on our side would be to exploit an SVO you could send us.
I saw in your video that you use a custom app. Would it be possible to add to it an on-demand recording feature? You could base it on our recording sample.
I’m sorry to ask this, I know it necessitates some work, but I think that is the quickest path to fixing this.
Also, does this happen to the camera over time when not using any particular module? Like, if you ran ZED Explorer over a long enough period of time, would it result in a black left image in the end? (Also, then it would be possible to easily record an SVO directly from ZED Explorer)
The most efficient way to investigate this on our side would be to exploit an SVO you could send us.
I will try to record an SVO the next time it happens; like I said it’s kinda rare so it could be tomorrow or it could be weeks, but I’ll post back here.
Also, does this happen to the camera over time when not using any particular module? Like, if you ran ZED Explorer over a long enough period of time, would it result in a black left image in the end?
I don’t know; we haven’t run the camera that way for a long period of time. I may not be able to get an answer to this question: Since we don’t know when it will happen next or how long it will be, it’s difficult to create a setup to test this.
We just ordered four more cameras, I will see if I can allocate one of them to testing with ZED Explorer. I don’t really have an extra machine at the moment to test them with though.
We think this might be the camera exposure handling in the camera firmware that is incorrectly behaving in some rare circumstances. The firmware would be in a bad state and would require a reboot to properly reset. The exposure would be at the minimum value for the left camera only, hence the non-black image.
We’re trying to find a way to add more security to detect this behavior by checking the camera’s internal state. In parallel, we’re working on adding a detection in the image_validity_check module, either pure image-based or ideally by detecting incorrect exposure time between left and right sensors.
We’re interested in an SVO file if you can record it, we’ll keep you posted if there’s a fix or other leads.
Edit: One way to verify this is to have a light in the field of view of the camera, this would still be typically visible with exposure at 0, something like a light bulb at a few meters should provide enough contrast
I think I managed to replicate this behavior by manually tweaking the exposure time and added a detection in the image_validity_check module. This will be part of the next release. We’re still working on finding the root cause but this should allow for a (hopefully) reliable way of detecting it.
I can send you a pre-release version if you’re interested.
@adujardin Sorry for the delayed reply; I was waiting for the issue to appear again (it did but unfortunately I once again missed the chance to record an SVO; but I will get one eventually).
At this point, we’re definitely interested in that pre-release; do you still have something available?
. If/when it happens again and I am able to collect that data, I will update here.
