We are running into a body tracking issue that’s becoming urgent for a client deployment, and I wanted to share specifics in case anyone has seen this or has guidance.
Environment
Camera: ZED 2i
ZED SDK: 5.1.1
Unity: 6.0.0 (6000.0.59f2)
ZED Unity plugin: body tracking pipeline using ZEDManager + ZEDBodyTrackingManager
Body model: HUMAN_BODY_ACCURATE
Body format: BODY_34
Body mode: UPPER_BODY (we only need shoulder elbow hand for gameplay)
Use case
Interactive game tracking 1 to 3 active players at once (dynamic)
We map hand position to a UI target using a shoulder elbow hand chain
Tracking works well when players are spaced, especially at ~1–3m
Failure occurs when a non player (or another person not intended to be tracked) stands very close directly behind an active player along the camera axis
Issue description
When a second person stands behind the target user (the user remains in front and visible), the target user’s arm joints become unstable:
shoulder elbow hand positions can jump or become incorrect for a few frames
sometimes the “front” user’s arm appears to be influenced by the person behind them
this breaks gameplay because our UI hand target teleports or drifts even though the front user is still clearly visible
It feels like an identity / association issue or multi-person pose ambiguity when silhouettes overlap, but we’re looking for guidance on what can be done within the SDK to improve robustness.
What we tried
Adjusting “Max Depth Range” in Advanced Settings (and bodyTrackingMaxRange from script). This did not improve the behavior in our 1–3m play space.
Keeping body tracking on HUMAN_BODY_ACCURATE, BODY_34, upper body.
We are already using smoothing on the visualization side, but the underlying joint positions still jump.
We can mitigate somewhat in game code by freezing or clamping joints when they spike, but we’d prefer to improve the raw tracking or correct configuration if we’re missing something.
Current SDK usage details
We subscribe to ZEDManager.OnBodyTracking and consume DetectedBody keypoints for shoulder, elbow, hand each frame.
We track multiple bodies but only use the joints from the active player(s).
The active player count is 1–3, set dynamically by the experience.
We’re using the ZEDBodyTrackingManager for body mode selection and tracking start.
Any guidance would be appreciated, especially if there are specific parameters or API calls we should be using for occlusion heavy multi-person scenarios.
@Myzhar Thanks for the reply and yes we have tuned the confidence settings for detection a lot. We also tried all the depth nodes except for neural. Neural caused body detection to stop.
I’m advances settings there is a max depth range which successfully blocks out people in the depth image passes a certain point but body tracking seems to be working independently of that setting.
We have it perfectly tuned so that users behind the play area won’t be detected. That part works well as it will not assign them skeletons but users. The issue is it seems to be blending non users and users if they stack behind each other in the image confusing the Zed to think the non users hand may belong to the user. So when our users are raising their hands the generated skeletons hand drops.
Is there another value you guys could expose to us or is it possible to have body detection work after max depth range has its pass?
Main improvement:
Try to use NEURAL_LIGHT or NEURAL as the depth mode.
You are using ULTRA, a deprecated mode with ZED SDK v5.1 that is still using the old Stereo processing instead of the new AI-based.
Setting the depth mode to Neural or Neural light would cause body tracking not to initialize for me. Simply switching back to Quality or Ultra and it initializes body tracking and detects users fine.
I think I see the issue. I didn’t catch the warning logs saying it needed to initialize. We will test with the Neural models and get back to you. Do you think Neural or Neural Light will provide greater results?
@Myzhar So after a day of testing there really wasn’t much of a noticeable difference in the tracking results on neural and neural light model. We noticed our issue only happens when someone of similar or larger height stands behind the user. It does not matter whether or not they are in the max tracking distance range, or a skeleton is detected at all.
I keep going back to the max depth range, but it seems that body tracking works independently of that. Is there some sort of value that you can expose to us in the sdk that can help with this?
What other suggestions do you have that may help us?
@Myzhar Thanks for the reply. Regardless of the max range we are seeing problems. Even when a spectator is outside of the max range but still behind another user the same thing happened. Without the SDK assigning a skeleton and recognizing them as a body. They are still able to conflict with existing users. Our depth is currently set at 4. Still having issues.
