We’ve been using ZED 2i on SDK 3.8.2 for about 7 months now onboard our drone for object detection and tracking through NVidia Deepstream.
In order to improve our exposure during object tracking, we began changing the Auto-Exposure/Gain ROI using setCameraSettings( sl::VIDEO_SETTINGS:AEC_AGC_ROI, ...) after every frame using the bounding box (+ 10% size buffer) as the ROI. This results in vastly better exposure, however the camera becomes unable to calculate depth >4m. We’ve tried using both sl::SENSING_MODE::FILL and sl::SENDING_MODE::STANDARD (and, by standard, taking the average of a 6x6 pixel area around the center of the bounding box), but neither perform any better. For now, we have to disable this feature in our code (fork of ZED-GStreamer), but would love to re-enable it as it helps our neural networks.
We can provide some MP4 files of the exposure in flight with and without the “exposure lock” and fork of ZED-GStreamer if desired through a private channel.
System:
Jetson Xavier NX
ZED-GStreamer (forked from master to add exposure lock ability)
Hi @penguin
we advise not using sl::SENSING_MODE::FILL for robotics applications where you need precise values for the depth estimation.
The FILL mode is a visual artifact introduced for AR/VR applications where you need a dense view of the depth map that is created by averaging close depth points, so adding artificial depth values that can be far from the real depth value.
What I do not understand is what you say here.
How is it possible that depth values cannot be estimated for depth >4m? Can you add more detail?
We have since moved away from sl::SENSING_MODE::FILL but just wanted to add that that’s what we had been using and tested to make sure there was no difference between that and sl::SENSING_MODE::STANDARD in our current issue
Our standard application is to grab the depth value from the center of the bounding box and then send it over to our flight controller. Ever since adding the code to set the AEC-AGC-ROI to be the bounding box, the depth values are almost always (>90%) outside of the valid depth range (0.3 - 40m) which we denote as “invalid”.
From ZED Depth Settings, it says “Avoid using the camera in very low-light environments as the camera uses color images for depth perception”. I’m not sure what qualifies as “very low-light”, but I do not think we are reaching this threshold as we are operating on bright sunny days.
I do see from Depth Stabilization that it fuses depth over multiple frames. At first, I thought this may be causing a problem as the exposure was changing rapidly at first which technically caused the RGB values of the image to change temporarily as well, but after a few seconds it was settling on an exposure level and the issue was persisting
@penguin I think that only pictures and eventually a short video can better clarify what’s the problem.
Also, the depth range is really high, unless you are using a camera with 4 mm focal length optics.
The 0.3-40.0m depth range was only for an external library as a “sanity” check when querying the depth map. We are currently using 0.3-30m but will be moving back to 0.3-20m.
I’ll send you a private message with video footage to see if that helps decipher the issue
I’ve performed a bit more in depth log review to see how reliable the depth has been in the past, and I think it’s starting to make a bit more sense.
We currently actively test on two different targets/platforms: Trailer and a Deck.
Trailer: Without our Exposure Lock, we get consistent depth (d) readings on our target in the range d < 4m; d > 10m. With Exposure Lock, we only get consistent depth (d) readings of the target in the range d < 4m. In both cases, there are other some valid depth values outside the listed ranges, but they are not consistent (i.e. invalid depth >90% of the time).
Deck: Without our Exposure Lock, we get consistent depth (d) readings on our target in the range 0 < d < 15m. We have only done one other flight with slightly better exposure (not object detection lock but only middle ninth exposure), and we see the same depth range but with less readings outside the normal range.
After reading through ZED Depth Settings, I think this may be a problem with the surface of the target/trailer/deck being texture-less and unable to calculate depth. From any sort of distance, the trailer appears texture-less in the image which would make it hard to calculate depth from. What confused me at first was that then “why do we get depth again when > 10m away?” I believe that this is because at ~10m away, the Trailer became small enough to use its physical edges to the ground along to calculate depth along with sl::SENSING_MODE::FILL being used. At around 4m, you can begin to make out a bit more textures on the surface of the Trailer which explains why the depth readings return at that range. After enabling Exposure Lock, the correct exposure on the Trailer removes a lot of the “noise” on the surface of the Trailer caused by incorrect/over-exposure without the Exposure Lock, hence why this issue came to light after enabling this feature. On our Deck Target, it is larger than the Trailer but has a lot more “extra” lines and features to the Deck that aid in the preventing the same texture-less surface issue we see in the Trailer. Like the Trailer, with better exposure (no Exposure Lock, just middle ninth exposure) the surface is “more texture-less” which was making it harder to calculate depth.
As such, we are looking to make these changes to our set-up to increase the consistency of the depth readings:
Change Depth Range from [0.3, 30m] to [1,20m] to improve performance, reduce jitter, and reduce inaccurate readings at larger distances
Change Depth Quality from PERFORMANCE to ULTRA for better depth, but this does decrease our neural network throughput (11 to 8 FPS due to increased GPU usage)
Re-enable our Exposure Lock using the object detection bounding box as the AEC-AGC ROI
Change Sensing Mode from FILL to STANDARD
There is one field/change I would like some input on though: Depth Stabilization. The link I posted above recommends it even for moving targets but says relies on the Positional Tracking capability which we have had issues with in the past. We can of course do tests with it turned on and off, but your input would also be valuable.
Thank you for the detailed feedback. Everything you said is correct, mostly regarding the textureless parts of the target.
NEURAL depth mode could improve this behavior, but FPS will be reduced for Xavier NX.
Regarding Depth Stabilization, it can surely improve the quality of the depth estimation, but it will not add information where it’s already missing. It only increases the stability of a depth pixel value by reducing the variance of the measure.
We changed the parameters above to their new values and verified that the depth we get is more constant and more accurate. Additionally, discovered that we were only changing the exposure for the LEFT camera which probably did not help the depth calculation since the two images would like vastly different in certain lightings.
We anticipate making a MR on the ZED GStreamer repository that contains support for updating auto-exposure/gain during runtime after we upgrade to ZED SDK v4 in the following weeks.