Help with Dual ZED 2i Cameras + LiveLink for AR Skeleton Tracking in Unreal

Software Unreal Engine
1

Hi everyone,

We’re currently working on a project using 2 x ZED 2i cameras and are aiming to achieve the best possible real-time skeleton tracking of multiple people in an outdoor environment. Our pipeline uses LiveLink to bring the tracking data into Unreal Engine, where we drive skeleton rigs for AR compositing.

The final output is a large outdoor screen that displays people walking past, with AR objects composited onto the live video feed, creating an immersive and playful installation.

:white_check_mark: Our Goal:

  • Use LiveLink with dual ZED 2i cameras for robust multi-person tracking.
  • Stream this data into Unreal Engine to drive skeleton rigs.
  • Composite AR objects onto the live video feed and display this in real time on a large outdoor screen.

:x: The Problem:

When we use LiveLink, we only get the tracking data, but no video feed comes through to Unreal. This forces us to use an additional video input and try to manually composite it ourselves – which is causing significant issues:

A. Camera Matching Issues
We’re struggling to match camera settings between the tracking and video feeds. Minor mismatches cause visual misalignment that breaks the illusion.

B. Camera World Positioning
We’re unsure how to determine the exact 3D world position of the camera when using LiveLink.

  • Does the camera spawn at 0,0,0?
  • How can we accurately align the camera feed with the tracked skeletons, especially in a dual-camera setup?

:question:Our Questions:

  1. What’s the best method for aligning an external video feed with the tracked data?
  • Any tips for matching intrinsics/extrinsics, FOV, or setting the correct camera transform?
  1. Is compositing in an external program like TouchDesigner or OBS a viable approach?
  • If so, how would we synchronize the skeleton tracking data with the video in those environments?

:pushpin: Context:

  • Installation requires wider tracking coverage, so using a single camera is less ideal.
  • We’d also like to use a higher-res video feed than what’s currently available via ZED in Unreal.
  • We’re aiming for crisp, real-time compositing to enhance the audience experience.

We really appreciate any help on this!

Thanks

2

Hi Emile,

You are using two cameras, so you must have calibrated the system using “ZED360”. That means the two cameras are located in a common coordinate frame where one camera, the first to be loaded, is defined as the world origin, its position will be (0, H ,0) with H being its height.

The 3D skeleton data is expressed in this coordinate frame as well. So if you place your virtual camera (or scene capture component) that displays the image at the position of the “real” camera, also expressed in this coordinate frame, everything should be aligned.

All the intrinsic parameters of our cameras are available via our API using sl::camera::getCameraInformation().

I’m sorry I can’t help you for the compositiing part. Maybe you can try to ask the TD team as they are also developing an integration for the ZED Camera for their software.

The Skeleton tracking data comes with a timestamp. You can use this information to sync it with the image.

1 Like