Hi everyone,
We’re currently working on a project using 2 x ZED 2i cameras and are aiming to achieve the best possible real-time skeleton tracking of multiple people in an outdoor environment. Our pipeline uses LiveLink to bring the tracking data into Unreal Engine, where we drive skeleton rigs for AR compositing.
The final output is a large outdoor screen that displays people walking past, with AR objects composited onto the live video feed, creating an immersive and playful installation.
Our Goal:
- Use LiveLink with dual ZED 2i cameras for robust multi-person tracking.
- Stream this data into Unreal Engine to drive skeleton rigs.
- Composite AR objects onto the live video feed and display this in real time on a large outdoor screen.
The Problem:
When we use LiveLink, we only get the tracking data, but no video feed comes through to Unreal. This forces us to use an additional video input and try to manually composite it ourselves – which is causing significant issues:
A. Camera Matching Issues
We’re struggling to match camera settings between the tracking and video feeds. Minor mismatches cause visual misalignment that breaks the illusion.
B. Camera World Positioning
We’re unsure how to determine the exact 3D world position of the camera when using LiveLink.
- Does the camera spawn at 0,0,0?
- How can we accurately align the camera feed with the tracked skeletons, especially in a dual-camera setup?
Our Questions:
- What’s the best method for aligning an external video feed with the tracked data?
- Any tips for matching intrinsics/extrinsics, FOV, or setting the correct camera transform?
- Is compositing in an external program like TouchDesigner or OBS a viable approach?
- If so, how would we synchronize the skeleton tracking data with the video in those environments?
Context:
- Installation requires wider tracking coverage, so using a single camera is less ideal.
- We’d also like to use a higher-res video feed than what’s currently available via ZED in Unreal.
- We’re aiming for crisp, real-time compositing to enhance the audience experience.
We really appreciate any help on this!
Thanks