# Multi camera point cloud fusion using room calibration file

I want to fuse depth point clouds from multiple cameras in real time.

It is my understanding from other forum posts that the fused point cloud model is still not yet ready for public release (as an aside, when is the ETA on that?).

So I’m thinking that I can do it by using a room calibration file produced by Zed360 to transform the point clouds from each camera into the same unified space.

Firstly, is this a reasonable approach?

And also: The docs say the rotation in the room json files is in radians, but they don’t specify in what order the rotations should be applied or much else about the transformations. So I have a few questions:

• Are the three rotations XYZ, respectively?
• What order should the X, Y, and Z rotations be applied in?
• Are these the rotations and translations to get from the camera coords to the unified space, or is it the inverse (from the unified space to the camera coords)?
• Should I retrieve the point cloud in “camera” space or “world” space?

Basically, I’ll need to construct a matrix M so that unified_point = M x camera_point, I just need a little more info about the transformations in the file.

I did look at the example at https://github.com/stereolabs/zed-sdk/blob/master/depth%20sensing/multi%20camera/python/multi_camera.py but that example doesn’t actually seem to transform the points into the unified space or read the room calibration file, it just kind of blends all the cameras’ points into an uncalibrated overlapping blob.

Thanks!

Hi,

Indeed, you cannot fuse raw point cloud right now. I have no ETA on this.
However, you can fuse the fused point cloud from our spatial mapping module. Check out this : https://github.com/stereolabs/zed-sdk/tree/master/spatial%20mapping/multi%20camera/cpp

Now, about what you are trying to achieve. Theoritically the approach is reasonable, we’d start like that. But if we did not achieve it yet, it is because it will be quite difficult even with a perfect room calibration.

The rotations are Euler Angles.
In the file the vector is XYZ,
For you matrix M you write
M.setRotationVector(sl::float3(X,Y,Z));

(values are en MILLIMETER et COORD_SYS IMAGE)

1 Like

Well I’ve loaded the room calibration file and I’ve switched over to using Unity to display point clouds, transformed by the calibrated positions, but the results don’t seem to be well aligned. In particular, the translation seems off.

In Unity I’m loading the calibration like this, into game objects that contain ZED Point Cloud Managers (using Newtonsoft JSON to parse the calibration file):

``````        JObject room = JObject.Parse(roomCalibration.text);
JObject camera = (JObject)room[cameraSerial.ToString()];
JObject world = (JObject)camera["world"];
JArray rotation = (JArray)world["rotation"];
JArray translation = (JArray)world["translation"];

float[] r = rotation.Select(v => (float)v * 180.0f / Mathf.PI).ToArray();
float[] t = translation.Select(v => (float)v).ToArray();

transform.SetLocalPositionAndRotation(
new Vector3(t[0], t[1], t[2]),
Quaternion.Euler(r[0], r[1], r[2])
);
``````

But everything seems off in translation by about 1 meter in Y.

I’m not really sure why, the alignment seemed OK in Zed360.

I’m aligning a zed2i with a zed mini.

Is there any reason why the room calibration transformations might not be the same for point clouds vs. skeletons?

I haven’t been able to check if the calibrations line up for skeletons yet because I’m having a lot of problems understanding the Unity body tracking API. So I’m not sure if I’m doing something generally wrong vs it being a problem unique to the point clouds.

I’ve disabled position tracking and initial position estimation on the cameras to keep it from interfering with the room calibration transformations but I’m also not sure if that’s the right thing to be doing.

In any case I’m having a lot of problems using the Zed360 room calibrations to line up point clouds between these two cameras, and I’m kind of out of ideas. Any advice would be helpful.

Thanks

Hi,

The camera calibration is not related to skeletons or point cloud, it’s just positions and angles. Theoretically, you can measure the distance between the cameras yourself and verify (or even build) the calibration file.

Then again, merging skeletons is quite easy compared to point cloud. Merging point clouds is very, very difficult.