Body Tracking elbows require Z depth?

Software Unity
1

This is a weird one, but I’ve noticed it consistently. Working in SDK 4.2, Unity 2022.3 with Body Tracking HUMAN_BODY_ACCURATE / BODY_38 / FULL, I can be in a straight-on H-pose at acquisition and the tracked skeleton’s arm will be angled straight from shoulder to wrist with a locked elbow. I turn my arm(s) so my elbows are still 90 degrees but my hands face the camera, and suddenly the skeleton begins tracking my elbows correctly, and going back to the straight-on H-pose is fine (modulo the usual imprecision with joint placement). I would think that a stereoscopic inference system would be able to get adequate info from a clear 2D right-angle, so it’s odd (buggy?) that it can’t parse accurately until it sees the bone straight-on…

(Ignore all the fancy masking, alignment, and avatar fitting stuff I’ve added to the SkeletonHandler; the core skeleton generation is still SDK-vanilla, it’s just easier to show it 1:1 with an isolated human subject)

armbug1
armbug11920×1164 144 KB

armbug2
armbug21920×1164 143 KB

armbug3
armbug31920×1164 145 KB

image
image465×1058 58.9 KB

2

Hi,

I think the issue is not coming from the detection itself but to the postprocessing we apply to it called body_fitting.

Unfortunately, the current implementation of the body fitting can create this kind of errors where the arms are “stuck” in certain positions.

3

The general documentation seems to imply that “fitting” is a 3rd tier of performance. Is there a way to toggle it within the Unity SDK as a test, or is it integral to the Body Tracking Module as provided?

4

After twiddling several settings without success, it does (provisionally) seem like dropping the tracking module to HUMAN_BODY_MEDIUM precludes this joint lock. Is that consistent with what the tech is doing under the hood?

5

The fitting is necessary to have access to the joint rotations, otherwise only the positions are available. That’s why you can’t disable it in Unity.

6

I don’t think so, it’s probably a coincidence and this will not be consistent.