Update on multi-camera fused point clouds through the python sdk

Software Python
1

What is the latest on being able to get fused point clouds from multi-camera fusion?

2

Hi @areiner222
the ZED SDK does not provide this function.
You can use Open3D or PCL to fuse point clouds (see ICP methods).

3

Got it thanks for the update.

What’s the best way to get the camera poses in world coordinates with the floor as the origin for all cameras?

Also, I noticed that for the python sdk, when I check the coordinate_system for the init fusion parameters I see a unit of measure?

init_fusion_parameters = sl.InitFusionParameters()
init_fusion_parameters.coordinate_system
“<UNIT.MILLIMETER: 0>”

4

You can use an ArUco tag placed on the floor:

I recommend you read the API documentation to find all the available settings.

5

Got it thanks! - Is it possible to set_floor_as_origin in the multi-camera setting like you can in the single camera? I only real care the origin is somewhere on the floor plane.

Tracking that the coordinate_system is not in the API Docs - thanks for pointing that out. I was seeing it as part of the python api.

6

Sure, you can. The camera pose will be initialized with a transformation relative to the detected floor plane.

7

I’m still struggling with using ZED360 calibrated camera fusion to getting the individual camera poses in world coordinates with the floor plane as the origin.

My fusion configurations are:

{'32547899': {'input': {'fusion': {'type': 'INTRA_PROCESS'},
                        'zed': {'configuration': '32547899',
                                'type': 'USB_SERIAL'}},
              'world': {'rotation': [0.0, 0.0, 0.0],
                        'translation': [0.0, -1.3583216667175293, 0.0]}},
 '38970398': {'input': {'fusion': {'type': 'INTRA_PROCESS'},
                        'zed': {'configuration': '38970398',
                                'type': 'USB_SERIAL'}},
              'world': {'rotation': [-0.017936093732714653,
                                     -0.7102773785591125,
                                     0.010944350622594357],
                        'translation': [1.2649309635162354,
                                        -1.0974094867706299,
                                        2.080958127975464]}}}

The y-coordinate translations here seem like the heights of the cameras relative to the floor plane which is what I want.

Then, I initialize the individual senders with positional tracking enabled relative to floor plane:

pos_tracking_params = sl.PositionalTrackingParameters()
pos_tracking_params.set_as_static = True
pos_tracking_params.set_floor_as_origin = True
...
status = senders[conf.serial_number].enable_positional_tracking(pos_tracking_params)

I set up fusion with positional tracking enabled:

pt_fp = sl.PositionalTrackingFusionParameters()
fusion.enable_positionnal_tracking(pt_fp)

And now, after grabbing, if I get the individual sender positions (translation in meters and orientation in euler angles):

# translations
{32547899: array([ 0.01625276, -1.59621799, -0.020855  ]),
 38970398: array([-0.00505199, -1.29675305, -0.03655138])}

# euler angles (degrees)
{32547899: array([-2.67991906, -0.01380266,  1.56679449]),
 38970398: array([-15.28537084,   0.02192709,  -1.01254673])}

These translations and orientations also seem sensible to me as they are measured relative to the floor plane (I don’t know exactly how the calibrated translation of ZED360 works).

Then I get the camera orientations from the fusion module:

poses = dict(zip(list(senders), [sl.Pose(), sl.Pose()]))
err_poses = [
    fusion.get_position(
        poses[cam_id.serial_number], uuid=cam_id,
    ) for cam_id in list(fusion.get_sender_state())
]
trans_fused = {k: pose.get_translation().get() for k, pose in poses.items()}
euler_fused = {k: np.degrees(pose.get_rotation_matrix().get_euler_angles()) for k, pose in poses.items()}

# translations
{32547899: array([ 0.01625276, -4.31286192, -0.02085507]),
 38970398: array([ 1.27042639, -3.92048454,  2.04593015])}

# euler angles (degrees)
{32547899: array([-2.67994617, -0.01182548,  1.56668563]),
 38970398: array([ -1.74144734, -40.69967317,   3.86045878])}

These poses don’t seem to be in a world coordinate system with the floor at the origin… .
Any walkthrough / explanation would be much appreciated!

Update: I forgot to mention that I am using the IMAGE coordinate_system for reading the calibration file and initializing the cameras.

8

Any help here would be much appreciated!
Thanks again!

9

Hi @areiner222
the ZED360 tool provides the transformation of each camera with respect to one of the cameras of the system that is used as the “main” device.

You cannot reinitialize each camera position with the transform provided by the floor detection function because you will mess up the fusion configuration.

What you can do is to create your own calibration method and initialize the Fusion module manually by setting all the Fusion transforms with known information.
See FusionConfiguration, start_publishing, and CommunicationParameters.

Please note that this is not straightforward. I recommend understanding well how each parameter influences the configuration.