Orientation Error (Pitch, Roll) when Camera Straight Down

Hardware
, ,
1

TLDR: We are noticing that the orientation (roll, pitch, yaw) becomes “stuck” for “large” amounts of time (0.5+ seconds at times) as the camera nears a straight-down pitch / looking straight down. It appears that this is due to the “6 DoF visual-inertial stereo SLAM” algorithm used to compute orientation (with sensor fusion) being unable to accurately calculate the orientation at said angle. Without using the visual odometry algorithm (ie no depth), the orientation is correct (noticeable error < ~1 degrees). It would be great to be able to configure the ZED camera to only use the senors to calculate its pose. This would be especially helpful when operating the camera in visually sparse areas (ie: at high altitude, over fields, water, etc).

More info:
To elaborate, we fly the ZED 2i Camera on a drone attached to a servo allowing it to pivot freely in the pitch axis. As the camera points down (tracking a target), the pitch of the camera stays at a specific angle for up to a second before correcting itself. This is most notably happening for the pitch but has been noticed in the roll (“freezes” at the same time). Since we don’t yaw very often while this occurs, I have not been able to check if yaw observes the same behavior.

We launch the code/camera using ZED-GStreamer in a custom CPP ROS-Melodic Node with the following parameters:

  • stream-type=4 #LEFT + DEPTH
  • camera-resolution=1 #1080p
  • camera-fps=30
  • camera-image-flip=0 #NO FLIP
  • enable-positional-tracking=1
  • enable-pose-smoothing=1
  • enable-area-memory=1
  • enable-imu-fusion=1
  • coordinate-system=0 #IMAGE
  • depth-mode=1 #QUALITY
  • sensing-mode=1 #FILL
  • initial-world-transform-x/y/z = 0.0
  • aec-agc=1 #custom region also set but not important
  • aec-agc-roi-side=2

In order to prevent this behavior, we had to “disable” depth by setting the min/max distance to 0.3m and 1.0m (depth-minimum-distance=300.0, depth-maximum-distance=1000.0) from the default of [0.3, 20.0]m. After setting the depth range to [0.3, 1.0]m, there were no longer “freezes”/errors in the pitch. We were still able to verify this behavior occurs by holding the camera within 1m of the ground (grass field) and observing the same error (pitch got “stuck” at 60 degrees, then updated to 75 degrees, then jumped to 90 degrees while rotating the camera smoothly), but then when holding it about 2m up, there was no longer any error.

If you would like any other details, I would be glad to provide.

Specifications:

  • NVidia Jetson Xavier NX
  • Jetpack 4.6.0
  • ZED SDK 3.7.2
  • Zed-GStreamer (master/latest)
  • ZED 2i
  • Ubuntu 18.04 LTS
  • NVidia Deepstream 6.0
  • ROS-Melodic
2

Update:
“Disabling” depth did not prevent the errors in orientation when pointing straight down during flight. We observed errors of up to 40 degrees where the camera was reporting for ~2 seconds that it was 60 degrees when in reality it was 108 degrees (camera updated with a 48 degree jump).

We are forced to attach an external IMU to the camera and use its orientation (RPY) instead of ZEDs reported orientation. At least with this method, we can re-enable depth :slight_smile: so our tracking code can use it

It would be great to be able to access only the IMU inside the camera, configurable through ZedSrc properties.

3

Hi @penguin
thank you for reporting this particular case.
The R&D team noted your information and will use them for future developments.

4

Is there any update?

5

Hi @Jaewon,

We have not returned to this issue on our system but think it may actually be caused by our parameter set-up on the camera, specifically sl::SENSING_MODE::FILL. While I claimed we “disabled” depth by setting range to [0.3, 1]m, sl::SENSING_MODE::FILL creates a fully dense depth map which “should” fully populate the depth map (although did not seem to always be case). Combine that with using the lowest quality depth mode QUALITY, and we were probably confusing the sensor fusion algorithm with bad data.

We are currently in the progress of upgrading from ZED 2i to ZED X camera which claims to have a newer “high performance” IMU. However, we won’t be doing testing to validate changes for at least a month. If you are seeing the same behavior we original saw, I would suggest using the highest/most accurate settings first.