Is there any plan/possibility to get ArUco positions when using Fusion?

Software Unity
1

just as the title asks -
we’d like some ArUco positions alongside our skeletons in the room -
we’re currently using Fusion - I think our solution for now would have to be adding a couple more Zed Cameras (not being used by the Fusion system) - and using them to detect the arucos with the Unity SDK -
but obviously we’d prefer not to have to add more cams to the space.
Thanks

1 Like
2

Hi @Rosss
You don’t need additional cameras for this; you can reuse the same cameras already feeding the Fusion module.

The key point is that the Fusion API and ArUco detection operate at different layers, so you can run both in parallel:

1. ArUco detection runs at the single-camera level, not in Fusion.
The Fusion module aggregates body tracking data from each Camera instance, but it does not perform ArUco detection itself. ArUco detection is done per-camera on the image retrieved from each Camera object. So for every camera you already have publishing to Fusion, you can also grab the left image and run the detector on it.

2. Use the same calibration Fusion already knows.
Each camera’s pose in the fused world frame is defined in your Fusion calibration file (the one produced by ZED360 or your custom calibration). When you detect an ArUco marker with one camera, you get the marker pose in that camera’s reference frame. You can then transform it into the Fusion world frame using the same World <- Camera transform stored in the calibration. This is exactly how skeletons end up in a common frame, so your markers will land in the same coordinate system as the bodies.

3. Detection options:

  • In the C++/Python SDK, there is an ArUco sample (zed-aruco) that shows how to retrieve the left image, undistort using the camera’s intrinsics from getCameraInformation(), run cv::aruco::detectMarkers + estimatePoseSingleMarkers, and build the marker pose. You feed it the per-camera intrinsics and your known marker size.
  • On the Unity side, you can retrieve the camera image and run an ArUco/OpenCV detection in C#, then apply the same camera-to-world transform.

So the workflow is: for each camera already in your Fusion setup, retrieve the image, detect markers, compute the marker pose in camera frame, then apply the camera’s world transform from the Fusion calibration. The result is marker poses expressed in the same world frame as your fused skeletons, with no extra hardware.

One caveat: marker detection quality depends on resolution and viewing angle, so a marker only contributes when it is clearly visible to at least one camera; you won’t get “fused” marker poses across cameras out of the box, but you can pick the best observation or average yourself if multiple cameras see the same marker.