Improving the Accuracy of Motion Capture Using UnrealEngine's LiveLink

Software Unreal Engine
1

Hi,

Currently, I am conducting to use ZED 2i to capture the motion of a single human dancing.
I have prepared a PC for motion capture, and using LiveLink to send the capture information to UnrealEngine running on another PC.
The code running on the motion capture PC is a C++ build of the code presented on LiveLink’s GitHub.

However, I am dissatisfied with the following points.

  • The accuracy of capturing the wrist is particularly poor.
  • When making intense movements, the capture may not be completed in time.
  • The motion appears to be captured, but the whole body floats in the air.

To solve this problem,

  • What settings in ZEDLiveLinkConfig.json would be effective in our situation?
  • What would be the ideal background for the person being captured? For example, would a solid green background be preferable?
  • What aspects of capturing using multiple ZEDs with the fusion API are expected to improve the accuracy of capturing? Will it solve the problem we are facing?

The ZEDLiveLinkConfig.json I am using is as follows. (This is just a sample.)
—————
{
“StreamingParameters”:
{
“connection_type”: “MULTICAST”,
“ip”: “230.0.0.1”,
“port”: 2000,
“send_bodies”: true,
“send_camera_pose”: true
},
“InitParameters”:
{
“input”: “USB_ID”,
“input_path”: 0,
“resolution”: “HD720”,
“fps”: 60,
“depth_mode”: “ULTRA”,
“grab_compute_capping_fps”: 0
},
“PositionalTrackingParameters”:
{
“set_as_static”: true,
“set_floor_as_origin”: true,
“enable_pose_smoothing” : true,
“enable_area_memory”: false
},
“BodyTrackingParameters”:
{
“detection_model”: “HUMAN_BODY_ACCURATE”,
“body_format”: “BODY_38”,
“confidence”: 50,
“max_range”: -1,
“minimum_keypoints_threshold” : -1,
“skeleton_smoothing”: 0.2
}
}

—————

Thank you in advance.

2

Hi,

Could you tell me which version of the ZED SDK you are using please?

Also, for accurate wrist detection, you might need to get quite close to the camera, do you have a rough idea on how far you were during your tests?
I can add that being fully visible (from foot to head) can be quite important as well.

Best,

Stereolabs Support

3

Hi,

Thanks for your reply.

I am using SDK version 5.02.

I would like to try detecting the wrist again by moving closer to the camera.

To improve the accuracy of detection of the whole body including the wrist, is it correct that the tip of the head should be at the top of the camera’s angle of view and the tips of the toes at the bottom?

Also, I would like to know if you have any tips about recommended backgrounds for better detection.

Thanks.

4

Yes, it is recommended to have the whole body in the field of view of the camera.

For the background, I’d avoid a flat wall with no texture as it can degrade the depth estimation.
But I think the most important is the lighting condition. In a dark environment, the accuracy will be greatly reduced.

Stereolabs Support

5

Thank you, your replies are very helpful!

I would like to ask a few additional questions.

  • We would like to capture intense motion such as dancing. Which parameters in ZEDLiveLinkConfig.json should we adjust for capture accuracy?

  • Is capturing with multiple ZEDs using the fusionAPI effective for capturing intense motion such as dancing? If so, how many ZEDs should be used?

Thanks.

6

Having multiple cameras will mostly help if you wish to cover a bigger space or keep a consistent accuracy overall (especially against occlusion), but will not help a lot to track fast movements.

The only parameter that can help is the frame rate of the capture. I’ll suggest using the 720p/60 fps combo to get the highest FPS possible while keeping a decent resolution. If you are really close to the camera, you can even try to use VGA/100fps but the quality will decrease.

Stereolabs Support

7

Would parameters like “confidence” or “skeleton_smoothing” be helpful to eliminate the overall breakdown?

Also, would it be better to use multiple cameras to improve accuracy when performing movements such as dancing with arms behind the body?
In that case, would it be reasonable to use two cameras capturing from the front and from the side?

8

The confidence threshold will not improve the accuracy of the tracking but can indeed remove some of the really bad ones.
Increasing the smoothing value will reduce the jitter but will add some latency to the output data.

Yes, in that case, adding another camera can help. If possible, I’d put it at the opposite side of the room, facing the first camera.

Stereolabs Support