We’ve been using ZED X Mini cameras on our robot for perception, including depth estimation. Recently we began noticing that the depth data pulsates significantly when the robot is in motion, making the depth practically unusable. We’ve tested using all three NEURAL depth modes and various levels of depth stabilization, but nothing has resolved the issue.
We’re currently using ZED SDK 5.1.2, and we interface with the camera using the ZED C API (from the 5.1 release, in case that matters).
The ZED X Mini in the SVO is mounted on the humanoid robot’s torso, angled down to see the terrain. We have another ZED X Mini in the head, which experiences the same depth quality issue. I can find an SVO from the head camera’s perspective as well if it’s necessary. They’re both connected to a Jetson AGX Orin.
Interesting. We’ll try again with fixed exposure and get back to you with the results.
In the meantime, I found an SVO from the head ZED X Mini. The same wobbling/pulsating depth is visible when the robot is in motion. Seems that even slight shakes of the head cause wobbling. I’ve also included a screen recording of what I see in the ZED Depth Viewer, just to be sure we’re looking at the same issue.
Are you using Depth Stabilization?
The default value is 30, that is good for dynamic conditions, but in your case the dynamic of the system is very quick, so you could try to lower this value.
We recorded two more SVO files for you to look at. Both of them have the exposure setting fixed to 75%.
One of them has the depth stabilization at 1, and the other one has it at 0. Both of these SVOs depth is not very usable.
You can see in the fixed-exposure-walking.svo2 that the point cloud is moving significantly in the Z axis. Making it unusable.
Things seem to be better in the fixed-exposure-0-depth-stab-walking.svo2 but not by much. The depth is still moving in the Z axis, making it not usable for terrain perception and footstep planning.
Does anything else come to mind that might cause this?
I am observing an inconsistency regarding the Depth Stabilization setting in the ZED SDK. While I understand this is a temporal filter that can be adjusted during SVO playback in the ZED Depth Viewer, I am seeing a noticeable difference in depth quality between SVOs recorded with stabilization ‘On’ versus ‘Off.’ Since SVOs are intended to store raw sensor data, why does the state of the stabilization toggle during the recording phase impact the output quality, and how does it differ from applying the same setting during post-recording playback?
I’ve reported this behavior to the ZED SDK together with your useful SVO files.
They will analyze all the provided information to try to understand if the problem is with your camera or if there’s something to be fixed in the Depth Processing when the device is used in similar conditions.
In the meantime, we are in the process of doing deep dive into all the parameters the ZED SDK provides that might affect the depth quality. We’ll keep this thread updated as we find out more.
As of now, we realized that self calibration was enabled, and might’ve affected the results we got between SVOs with depth stabilization enabled and disabled. Either way, we definitely see the depth pulsating regardless of the depth stabilization strength, so it must be something else.
We also tried swapping out the head ZED X Mini for a newer one to check whether the hardware had degraded over time. The new ZED X Mini gave us the same pulsating depth, so we think the hardware is still good.
We’ll play around with exposure time next since the pulsations in the depth seem to be in sync with motion blur in the images. Please let us know if any other parameters that we should test come to mind!
Can you explain what you mean here? Self calibration works at runtime and it’s invisible to the user, unless you retrieve the extrinsic calibration values and you use them in your processing.
I mean to say that we have been setting the SL_InitParameters#camera_disable_self_calib parameter to false. If I understood correctly, that enables self calibration and will re-calibrate the camera when we open it. We’ve been under the assumption that the depth quality may differ between calibrations. Is that correct?
We do query the SL_CalibrationParameters for intrinsics (focal lengths and principal points) for rendering the point cloud in our UI, though I assume that wouldn’t affect the depth data when played back in the ZED_Depth_Viewer.
Temperature variations, shocks, and continues vibrations can affect the calibration. So you can also manually trigger a new extrinsic calibration once in a while.
You can read more here:
Ok, we can look at that but we believe we found our issue.
We use a Connect Tech Forge carrier board to run our Jetson and ZED cameras. It seems the board support package version we were using was somehow messing up the depth of the ZED X Mini.
By upgrading the board support package on our carrier board the pulsating depth during motion has been fixed.
Thank you for the help throughout our debugging process!