NEURAL PLUS point cloud curves on flat cardboard surfaces (static scene) - best config to avoid it?

Software
1

Hi,

I’m capturing static scenes with ZED X cameras and using the point cloud for cardboard box detection and pose estimation. My problem is that flat surfaces do not come out flat: cardboard box faces bow outwards in the point cloud, which then tilts the estimated box pose. It is worst on small, low-texture faces and when a box is partially occluded.

curved_pointcloud_crop
curved_pointcloud_crop1150×640 210 KB

The box the arrow points at is a plain, flat-sided cardboard box sitting on the floor. Viewed nearly edge-on, its front face is clearly bowed out towards the camera in the middle instead of forming a straight vertical line, and the surface sags away at the edges. Nothing in the real scene is curved. Both clouds are ZED X in NEURAL PLUS.

What I’ve already tried

  • Disabled depth stabilization (depth_stabilization: 0) - this does reduce the curving, but doesn’t remove it.
  • Lowered the confidence threshold to drop the bad points - this doesn’t really help: I lose good points before I lose the curved ones.

My current configuration (ZED ROS 2 wrapper) -

general:
  grab_resolution: "HD1200"
  pub_resolution: "NATIVE"

depth:
  depth_mode: "NEURAL_PLUS"
  publish_point_cloud: true
  xyz_only_pointcloud: true
  depth_stabilization: 0
  depth_confidence: 50
  depth_texture_conf: 100

pos_tracking:
  pos_tracking_enabled: true
  set_as_static: false
  imu_fusion: false

video:
  auto_whitebalance: true
  whitebalance_temperature: 42
  saturation: 5
  sharpness: 5
  gamma: 1

Questions

  1. For a static scene and a static camera, what is the recommended depth configuration? Since I don’t need temporal filtering at all, is depth_stabilization: 0 correct, or is there a better approach - e.g. grabbing several frames and averaging?

  2. Is depth_texture_conf: 100 a bad choice for low-texture surfaces? My targets are plain cardboard with very little texture. Would lowering this filter out exactly the unreliable regions that are curving, and what value would you suggest as a starting point?

  3. Is this curving expected behaviour of the neural depth models on textureless planes? If so, is NEURAL_PLUS actually the best choice here, or would NEURAL, or even a non-neural mode, give flatter planes on this kind of surface?

  4. Does pos_tracking_enabled: true with set_as_static: false affect depth at all when the camera is not moving? Should I set set_as_static: true, or disable positional tracking entirely for a fixed camera?

  5. Is there a recommended way to reject curved/bent regions that works better than the confidence threshold? Anything exposing per-pixel depth uncertainty that would let me drop the bad regions without also losing valid points?

Any guidance on the best overall configuration for accurate, flat point clouds on static, low-texture scenes would be much appreciated. Thanks!

2

Hi,

Thanks for the very clear write-up, the config and the annotated cloud made this much easier to reason about. Taking your questions in order:

  1. On a fixed camera I would not leave this at 0. Temporal stabilization averages out exactly this kind of noise, and in our tests raising it to 30-60 clearly flattened planes. Worth A/B testing on your scene.

  2. Best left at 100. depth_texture_conf removes depth in uniform areas, so on plain cardboard it tends to delete the face itself rather than the bad points, much like what you saw with the confidence threshold.

  3. In some instances yes. Textureless surfaces are a notorious case for stereo matching, neural models included: with no pattern to match, the result is driven by the model’s smoothness prior rather than by real evidence, which shows up as a gentle bow rather than obvious noise.

  4. No, positional tracking does not affect depth while depth_stabilization is 0, so that one is not your culprit. If you do enable stabilization, set_as_static: true is the right choice for a fixed camera.

  5. Unfortunately the confidence map will not separate these points either, as the bow is a systematic bias rather than low-confidence noise. Rejecting at fit level works better, for example a RANSAC inlier ratio, or constraining the fit with your known box dimensions.

NEURAL_PLUS is the right choice, so no need to change that. If you can share the working distance and an SVO2 of the scene, I am happy to dig further.

Best regards,

Tristan