Jitter in skeleton tracking and depth roi not effective (4.0.4)

Software C++
1

Hi,
I have notice that sometimes the skeleton data, some keypoints have x,y and z at value 0,
like a bunch of keypoints like the nose and others althought the rest have normal values.

Is there something to do with the body confidence threshold?

Is it normal and can someone explain to me how thath works.

I am using the skeleton data as a controller for a game and this causes the movement to jitter sometimes I guess.

Is there a know list of the keypoints that is more difficult to get that others?

Besides Even though I have ROI the depth, I still get some skeleton detected that are supposed to be out of range. Those skeletons are behind the main skeleton which is within range.

Would appreciate some help on those issues

Thanks

2

Hi @robotpapier,

  • What body model are you using? (38-34-18)
  • Are we talking about relative or absolute position?
  • Do you mean that the data jumps to the 0 value, inducing jitter, or that it’s just at 0 sometimes?
  • Is the fitting enabled?
  • Is that always the same points that are set to 0?

The keypoint confidence threshold will give the nan value to the unsatisfying keypoints. If you enabled the fitting, all keypoints should have a valid value though (inferred for some).

The keypoints that are more easily obstructed, like on the head, are the easier to lose.

The detected skeletons outside of the depth should be located at the “infinite”. You might want to filter these out using your method of choice (based on their keypoint[0] position for example).

3

Hi thanks for your reply.

This is my config

    positional_tracking_parameters.set_as_static = true;

    body_tracker_params.enable_tracking = true; // track people across grabs
    body_tracker_params.enable_body_fitting = true; // smooth skeletons moves
    body_tracker_params.body_format = sl::BODY_FORMAT::BODY_38;
    body_tracker_params.detection_model = BODY_TRACKING_MODEL::HUMAN_BODY_ACCURATE;
    body_tracker_parameters_rt.skeleton_smoothing = 0;
    rt_params.measure3D_reference_frame = REFERENCE_FRAME::CAMERA;

I guess it is relative position, as the [0,0] of [x,y] is at the center of the image.

I mean that in my skeleton data sometimes indeed bunch of keypoints located at the head mostly with some others have x,y and z equals to exatcly 0.0 where as in the same data frame, others keypoints have the valid value.

Yes it seems to be always the sames points that are equal to 0 when the skeleton data is not completely giving valid values for all keypoints.

OK, I think I just ofund out something.
I was doing this in my bodyDataToJsonSimple function : :sweat_smile:

    res["position"]["x"] = isnan(body.position.x) ? 0 : roundToMaxDecimals(body.position.x / 1000);
    res["position"]["y"] = isnan(body.position.y) ? 0 : roundToMaxDecimals(body.position.y / 1000);
    res["position"]["z"] = isnan(body.position.z) ? 0 : roundToMaxDecimals(body.position.z / 1000);

So indeed as you said, it give nan for insatisfying keypoints.

So my new question is how to not include from the beginning (without filtering afterwards) a skeleton data frame, if there is nan values?

Do you have like an ordered list of more easily obstructed keypoints from the most to the less? So I will not tend not to use them ?

4

I was looking at the doc

  • Skeleton Smoothing: Amount of smoothing applied at the body fitting phase in the Body Tracking module of the ZED SDK. Bigger values mean more latency and smoother movements, and lower values lead to a crispier animation with less interpolation, which is better for quick movements.

Can my problem be solved is I set a higher value for smoothing?
Right now it is 0.

Besides if I don’t set the smoothing value, what is the default value that is being used?

5

So my new question is how to not include from the beginning (without filtering afterwards) a skeleton data frame, if there is nan values?
Do you have like an ordered list of more easily obstructed keypoints from the most to the less? So I will not tend not to use them ?

For now, you can only set the minimum_keypoint_threshold along the detection_confidence_threshold to filter this before getting them. A keypoint not detected or filtered by the confidence will give NaN, and if the minimum keypoint is not reached the skeleton will not be given.

I don’t have an ordered list to share with you, but you can take a look at the skeleton structures for the body models in the documentation, and test it on your side, to get a feel of what best sensed and what’s not. Well, what’s best detected in general is the torso area and the face, and the not-so-good detections generally happen with the hands (in body_38). Note that body_34 is fully based on body_18 detection.

Can my problem be solved is I set a higher value for smoothing?

You can try to set it to 1, it should reduce the jitter but it won’t fix the NaN values.

Besides if I don’t set the smoothing value, what is the default value that is being used?

The value is a parameter of a lot of things that we’re trying to format and maybe expose, as it’s different for each body model.

6

Hi @JPlou
Thanks for all those informations.
Now that I know that a skeleton frame can have NaN value, I did a postfiltering so it does’nt glitch my joint.

But I still get glitches and I think I know why.
I was testing my game and like 3-4 person passed behind me very far like 8 meters away.
My depth ROI is between 1 and 2 meters.
And as they passed, my joints started to jitter.

I think, when a lot of people passes behind even if out of ROI, it makes all bodies in in the screen (at least my most prominent one) less confident and then more likely to give NaN values.

What do you think?
Is it a know problem?

My confidence was on 30.

Now I put it to 80, but as I don’t have 3-4 people to test with me, I have to wait until some people passe again to be sure.

Do you have any information on this behaviour?

besides the skeleton smoothing I set to 0, the jitter has nothing to do with it I think :confused:

7

I don’t think we’ve tested this much, this feedback is important for us, as we’ve not witnessed this before.

When you say the joints jittered when people passed behind, did the ID of the skeleton switch? It should be visible by its color in a body tracking sample.

Setting the confidence high should filter the background detections, which may indeed fix the issue.
The foreground confidence should always be pretty high.

Are you using tracking, fitting, both, or neither? Also, which body model and detection model (fast, medium, accurate)?

8

Hi,
Yes I use tracking and fitting (true)
I don’t know if tracking loses track because I have modified the opencv drawing function to highlight only the skeleton that is the most in the center of the image and within my Z range.

With my filter, there is no jitter, so I guess the filter works well.
I’m almost sure, the main skeleton gets confused lot of people passes behind.

Not verified with bunch of people behind, but I think high confidence avoid this problem too.

Also I use BODY::38 with ACCURATE detection, and I noticed also that with ACCURATE detection, the 3D map is nto very unusable, too fuzzy and unstable unlike NEURAL, I wonder how detection other that NEURAL can be usable by the way :confused:

I use ACCURATE because I noticed the FPS is a lot better than NEURAL.
Maybe I should switch to neural to avoid Z measurement imprecision, that could explain why it detects body over the depth ROI i have set. But I will lose fps.

What do you think If I switch to BODY_18 ? will it be faster? When I tested at first I didn’t see much differences with my RTX3060, that’s why I stayed on BODY_38

9

Hey,

Switching to Body18 will be a bit faster, but as you said there isn’t much difference, it depends on the output you want.
You probably want to record an SVO of the problematic situation to test on your side, and share with us if you want us to take a look.