Issues with multi-camera fusion: only camera 0 correct, point clouds not fused (ZED SDK C++ example + ZED360 calibration)

Software C++
, , ,
1

Hi everyone,

I’m trying to run the official C++ “multi-camera fusion” sample on a Jetson Orin NX (16 GB) with four ZED X cameras connected via GMSL2. I calibrated all four cameras using ZED360 (following the official guide) and verified that in the ZED360 UI they are positioned at the same height (forming the corners of a rectangle) and all pointing toward the intersection of the diagonals.

When I launch the sample (with my own JSON config from ZED360), I see:

  • Only camera 0 (the reference camera in my JSON) rendered correctly.
  • Cameras 1–3 appear tilted/shifted incorrectly in the GLViewer.
  • The fused point cloud is not fused at all; instead I get four overlapping clouds (one per camera).

Here’s a screenshot showing the problem:

Screenshot from 2025-05-09 23-32-36
Screenshot from 2025-05-09 23-32-362068×1207 921 KB

What I’ve tried:

  • Verified JSON poses are correct (identity + 3 relative transforms)
  • Removed any Fusion SLAM calls (enablePositionalTracking + getPosition) to force pure static extrinsics
  • Ensured I apply each pose exactly once when drawing raw clouds (in that way, cameras are correctly drawn)
  • Still, each camera cloud remains in its own plane and the fused cloud is never combined (as you can see in the following screenshot)

Screenshot from 2025-05-09 23-03-30
Screenshot from 2025-05-09 23-03-302068×1207 940 KB

Environment:

  • Hardware: ZED Box Orin NX 16 GB
  • Cameras: 4× ZED X via GMSL2
  • JetPack: 6.0
  • ZED SDK: 5.0
  • Sample code: latest multi_camera_fusion example from GitHub

My questions:

  1. Am I missing a specific Fusion API call to retrieve the truly fused point cloud in world coordinates?
  2. Is there an extra step needed to tell Fusion “use these static extrinsics only, no SLAM drift”?
  3. Does anyone know why only camera 0 ends up correct and the others keep their local orientation?
2

Does anyone knows how to solve?

3

Do you know if some of the cameras are flipped?

If so, can you, in the ClientPublisher.cpp code, set the sl::InitParameters::camera_image_flip to sl::FLIP_MODE::ON or sl::FLIP_MODE::AUTO ?

Stereolabs Support

4

Unfortunately, nothing changes. It seems like all the point clouds are displayed in their own coordinate system and not fused at all.
I took the code in github as it is, i noticed it also in the multi_cam_multi_model_example, like it is a problem related to GLViewer.cpp file

5

I think you are running the sample that shows how to open multiple cameras at the same time, not the sample that demonstrates how to display the fused pc.

You need to run this one: https://github.com/stereolabs/zed-sdk/tree/master/depth%20sensing/fusion/cpp

Stereolabs Support

6

Ok, i’ll try just to make sure everything is correct!
zed-sdk/object detection/multi-camera/cpp at master · stereolabs/zed-sdk · GitHub i’m currently working with this code, is not written to perform object detection in 3D?

7

i got the same problem also with the viewer in this code!
Here’s my configuration file

{
“42844251”: {
“FusionConfiguration”: {
“communication_parameters”: {
“CommunicationParameters”: {
“communication_type”: “INTRA PROCESS”,
“ip_add”: “”,
“ip_port”: 0
}
},
“input_type”: {
“InputType”: {
“input”: “”,
“input_type_conf”: “42844251”,
“input_type_input”: “GMSL SERIAL”
}
},
“override_gravity”: false,
“pose”: “1.000000 0.000000 0.000000 0.000000 0.000000 1.000000 0.000000 -2.044726 0.000000 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000”,
“serial_number”: 42844251
}
},
“42979783”: {
“FusionConfiguration”: {
“communication_parameters”: {
“CommunicationParameters”: {
“communication_type”: “INTRA PROCESS”,
“ip_add”: “”,
“ip_port”: 0
}
},
“input_type”: {
“InputType”: {
“input”: “”,
“input_type_conf”: “42979783”,
“input_type_input”: “GMSL SERIAL”
}
},
“override_gravity”: false,
“pose”: “-0.997812 -0.013258 -0.064775 1.841314 -0.011876 0.999695 -0.021678 -1.963183 0.065042 -0.020861 -0.997665 6.697111 0.000000 0.000000 0.000000 1.000000”,
“serial_number”: 42979783
}
},
“46461707”: {
“FusionConfiguration”: {
“communication_parameters”: {
“CommunicationParameters”: {
“communication_type”: “INTRA PROCESS”,
“ip_add”: “”,
“ip_port”: 0
}
},
“input_type”: {
“InputType”: {
“input”: “”,
“input_type_conf”: “46461707”,
“input_type_input”: “GMSL SERIAL”
}
},
“override_gravity”: false,
“pose”: “0.233698 -0.054104 0.970803 -2.671591 0.030253 0.998372 0.048358 -2.158595 -0.971838 0.018068 0.234955 2.867991 0.000000 0.000000 0.000000 1.000000”,
“serial_number”: 46461707
}
},
“49392992”: {
“FusionConfiguration”: {
“communication_parameters”: {
“CommunicationParameters”: {
“communication_type”: “INTRA PROCESS”,
“ip_add”: “”,
“ip_port”: 0
}
},
“input_type”: {
“InputType”: {
“input”: “”,
“input_type_conf”: “49392992”,
“input_type_input”: “GMSL SERIAL”
}
},
“override_gravity”: false,
“pose”: “-0.072220 -0.029359 -0.996957 4.191905 0.021768 0.999282 -0.031005 -1.939853 0.997151 -0.023941 -0.071529 3.989908 0.000000 0.000000 0.000000 1.000000”,
“serial_number”: 49392992
}
}
}

There is some issue related to the configuration? As i said before, the only correct cam is the reference one (id 42844251).

8

I’ve tried almost everything, the fusion seems broken. If i subscribe only one cam the point cloud is displayed correctly, and each camera added destroys the point cloud. The config file is generated from ZED360, the cam are at the vertices of a rectangle looking at the centre…

9

I noticed that if the cam are not looking at the plane, it works correctly. Is there a way to support cameras oriented in that way?

10

Can you try to replace this line : https://github.com/stereolabs/zed-sdk/blob/master/depth%20sensing/fusion/cpp/src/main.cpp#L155
with : if(fusion.getPosition(pose, sl::REFERENCE_FRAME::WORLD, id, sl::POSITION_TYPE::RAW) == sl::POSITIONAL_TRACKING_STATE::OK) ?

Stereolabs Support