Hi @karinawilson
Welcome to the StereoLabs community
Thank you for the very detailed report; this level of information makes the analysis much easier.
objects in the fused cloud appear doubled/offset rather than as a single coherent shape, indicating an extrinsic calibration problem between the two cameras
Before chasing the extrinsics further, there is an important design detail of ZED360 that explains a good part of what you observed. As documented in the Useful knowledge about the Fusion API Calibration section, the optimizer only estimates the yaw of each camera; pitch and roll are taken from the camera IMU, and the floor level is estimated only if the ankles of the person are detected during the walk-around. ZED360 minimizes the distance between body keypoints, which are noisy at the centimeter level, so it is designed to produce a calibration good enough for skeleton fusion; it is not a metrology-grade extrinsic solver for point cloud registration.
This has two consequences for your case:
- Two runs on the same rig can converge to different solutions, because the cost function is weakly constrained if the person does not cover the whole shared volume at several depths. The jump you saw from ~17° to ~6.5° with a translation sign flip is typical of an under-constrained optimization, not of a moved rig.
- Any residual pitch/roll error coming from the IMU cannot be corrected by the optimizer, and a fraction of a degree of tilt is enough to produce visible doubling on a target a few meters away.
Step 1: validate each camera individually before fusing. Ghosting is not necessarily extrinsic. Open each camera separately in ZED Depth Viewer at the working distance and check a flat reference surface for bowing or a distance bias. If one of the two stereo calibrations has drifted (impact, strong thermal cycling, glass in front of the lens), no extrinsic transform will ever align the two clouds, and ICP fitness will stay low. You can restore the factory calibration from ZED Explorer, and use ZED Calibration only if you have a specific reason to override the factory data. Also use DEPTH_MODE::NEURAL_PLUS for this kind of geometric work, see Depth Modes, and set a conservative confidence_threshold so that unreliable edge points do not look like a second surface. One more thing worth testing: set InitParameters::camera_disable_self_calib = true on both cameras, so the self-calibration performed at each open() cannot slightly change the stereo geometry between your calibration session and your capture session.
Step 2: check the convention used to inject your own extrinsics. This is the most frequent cause of a “stable solve, wrong result”. As described in the Fusion configuration files section, the pose stored in the JSON must be expressed in meters and in COORDINATE_SYSTEM::IMAGE; the SDK read/write helpers handle the conversion to the coordinate system you requested in InitParameters. If you parse the JSON yourself and apply the pose to clouds retrieved in RIGHT_HANDED_Y_UP, you get exactly the kind of axis permutation and sign flip you are describing. The Coordinate Frames page has the reference conventions.
Step 3: on the marker approach. A 16 mm / 1.3° sample-to-sample spread is repeatability, not accuracy; a single 160 mm marker seen from a few meters gives a poorly conditioned rotation estimate and is subject to planar pose ambiguity, so the bias can easily be an order of magnitude larger than the spread. A ChArUco board, or several markers spread over the working volume, will constrain the solution much better. Please also check that you are feeding solvePnP with the rectified intrinsics and zero distortion (camera_configuration.calibration_parameters, not ..._raw), since you are working on rectified images.
Practical path that usually works for a static scene like yours: use ZED360 only to get a coarse initialization, then refine with ICP on the static test article itself, and write the refined pose back into the Fusion JSON. With a coarse init available, the ICP max_correspondence_distance should be a few centimeters, not meters; a fitness of 0.06 usually means the initial guess is far off, the units differ (millimeters against meters), or the two clouds barely overlap. Also make sure both clouds are cropped to the shared volume before running ICP, otherwise the non-overlapping background dominates the correspondence count.
Finally, please keep in mind that two ZED 2 cameras on USB are not hardware synchronized. This is not an issue for a static article, but it does add noise to the calibration walk-around; walking slowly, as recommended in the ZED360 procedure, is important. If you plan to grow this rig, the ZED X family on GMSL2 supports hardware synchronized capture through the ZED Link capture cards, which removes that error source entirely (store page).
If you can share the ZED SDK version, the camera firmware versions, the ZED360 JSON file, and two short SVO2 recordings of the scene (one per camera), we can check the geometry on our side and tell you whether the residual error is extrinsic or per-camera depth.