I want to fuse depth point clouds from multiple cameras in real time.
It is my understanding from other forum posts that the fused point cloud model is still not yet ready for public release (as an aside, when is the ETA on that?).
So I’m thinking that I can do it by using a room calibration file produced by Zed360 to transform the point clouds from each camera into the same unified space.
Firstly, is this a reasonable approach?
And also: The docs say the rotation in the room json files is in radians, but they don’t specify in what order the rotations should be applied or much else about the transformations. So I have a few questions:
Are the three rotations XYZ, respectively?
What order should the X, Y, and Z rotations be applied in?
Are these the rotations and translations to get from the camera coords to the unified space, or is it the inverse (from the unified space to the camera coords)?
Should I retrieve the point cloud in “camera” space or “world” space?
Basically, I’ll need to construct a matrix M so that unified_point = M x camera_point, I just need a little more info about the transformations in the file.
Now, about what you are trying to achieve. Theoritically the approach is reasonable, we’d start like that. But if we did not achieve it yet, it is because it will be quite difficult even with a perfect room calibration.
The rotations are Euler Angles.
In the file the vector is XYZ,
For you matrix M you write
M.setRotationVector(sl::float3(X,Y,Z));
Well I’ve loaded the room calibration file and I’ve switched over to using Unity to display point clouds, transformed by the calibrated positions, but the results don’t seem to be well aligned. In particular, the translation seems off.
In Unity I’m loading the calibration like this, into game objects that contain ZED Point Cloud Managers (using Newtonsoft JSON to parse the calibration file):
JObject room = JObject.Parse(roomCalibration.text);
JObject camera = (JObject)room[cameraSerial.ToString()];
JObject world = (JObject)camera["world"];
JArray rotation = (JArray)world["rotation"];
JArray translation = (JArray)world["translation"];
float[] r = rotation.Select(v => (float)v * 180.0f / Mathf.PI).ToArray();
float[] t = translation.Select(v => (float)v).ToArray();
transform.SetLocalPositionAndRotation(
new Vector3(t[0], t[1], t[2]),
Quaternion.Euler(r[0], r[1], r[2])
);
But everything seems off in translation by about 1 meter in Y.
I’m not really sure why, the alignment seemed OK in Zed360.
I’m aligning a zed2i with a zed mini.
Is there any reason why the room calibration transformations might not be the same for point clouds vs. skeletons?
I haven’t been able to check if the calibrations line up for skeletons yet because I’m having a lot of problems understanding the Unity body tracking API. So I’m not sure if I’m doing something generally wrong vs it being a problem unique to the point clouds.
I’ve disabled position tracking and initial position estimation on the cameras to keep it from interfering with the room calibration transformations but I’m also not sure if that’s the right thing to be doing.
In any case I’m having a lot of problems using the Zed360 room calibrations to line up point clouds between these two cameras, and I’m kind of out of ideas. Any advice would be helpful.
The camera calibration is not related to skeletons or point cloud, it’s just positions and angles. Theoretically, you can measure the distance between the cameras yourself and verify (or even build) the calibration file.
Then again, merging skeletons is quite easy compared to point cloud. Merging point clouds is very, very difficult.
