Errors with Unity Live link and Multi-Camera Fusion Tracking

Software
, ,
1

Hello,

We are currently using 4 Zed 2i cameras with Fusion API calibrated through the ZED360.
The tracking looks fine when we have 1-2 people in the room, but it would start running into issues as we have more people.

Here are the list of the issues we are experiencing:

  1. From the live link app, the tracking looks extremely inaccurate when we have 4 or more people. The tracking result seems jumping all over the places(Please find attached video for reference)

  2. The console for the live link app would output error message regarding the UDP messages and Unity will thus not be able to receive any updates

Send Failed(sendto()): Domain error
Send Failed(sendto()): No error

Here’s the computer spec:

  • CPU: 13th Gen Intel® Core™ i7-13700
  • Memory: 16 GB, 2 x 8 GB, DDR5, 4400 MHz
  • GPU: NVIDIA® GeForce RTX™ 3070, 8 GB GDDR6, 3 DP, HDMI

And for your reference, here’s a video of the Live link screen when there are 12 people in the space: Dropbox - IMG_6617.MOV - Simplify your life

We would really like to get some help to get a better tracking experience.

Best

2

Hello @sil, welcome to the forums

First, let me say that we recently identified some issues with the body tracking performance in the latest SDK versions and are working on fixing them. You should not get 6 fps with your configuration, even using Body38 accurately (which is probably what you’re using).

The low and unstable FPS probably causes the inaccuracy and jumping.

To improve the FPS, you can try to use fewer cameras, effectively running fewer instances of the SDK. You can use a less taxy body detection model, like HUMAN_BODY_FAST or MEDIUM. You could also use a less accurate depth mode, but it should not improve a lot.

To improve the stability of the FPS, I would first see how much fps you can get with 4 cameras running, then set the grab_compute_capping_fps InitParameter of each sender to the rounded value just below this. Let’s say you get 17 fps on each camera, set each camera fps to 15. 11, to 10, etc.

We’re working on these performance issues and will release a fix as soon as we can. You should at least get stable 15 fps then, and the Fusion will work way better. Sorry for the inconvenience.

Edit: it’s grab_compute_capping_fps that you need to change, not camera_fps (which you can lower too but won’t have the expected effect).

3

Thanks for your reply.

We have made some progress since we posted this question.
The tracking is much improved after we dropped one of the cameras - setting the depth mode to a lower quality also significantly helps with tracking more people.

However, our biggest challenge right now is to get the tracking data to Unity. We ran a test with 12 people in the room, on the fusion live software, the tracking result seems stable, but the data doesn’t seem to be transferred properly.
As I mentioned in the original post and shown in the video, the following error message will start ocurring after 7-8 people in the room.

Send Failed(sendto()): Domain error
Send Failed(sendto()): No error

My assumption is that there’s too much data being sent and UDP is not able to handle it. I wonder if that’s something the dev team have encountered?

4

Hi @si, that’s good news (well the best should be when the wore issues are fixed on our side, but still, nice!)

The UDP problem is a possibility, I don’t think we tried the Unity Live Link with that many people.
I’ll add that to our to-do soon list. Thanks for the feedback!

5

Hello, @JPlou

We are also setting up 4 x Zed2i cameras using fusion API calibrated through the ZED360.

we were easily able to calibrate cameras and Track 4 to 7 persons in our 17ft x 32ft room.
we are using the Unity live link to send data to our Unity application.

Till the seventh person, it is working fine. As it detects the eight-person our unity application freezes.
When that eight-person leaves out of the camera detection, it returns like clockwork every time.

our speculation was the same, too much data was being sent to UDP and it was not able to handle it properly.

So, We have tried different Skeleton like only the upper body to reduce data transfer to see if any improvement but no improvement at all.

we are also facing the skeleton jumping arround here and there. we have tried to smooth that motion.

we have tried these with 4 cameras as well as 3 cameras.

In 4 cameras we are getting 15fps when 7 or 8 people are there. but again it freezes in the Unity application.
Same thing we tried with 3 cameras performance increased but the freezing issue is still there.

we are also getting these errors when this happens. Sometimes these errors start when we launch the live link app.

Send Failed(sendto()): Domain error
Send Failed(sendto()): No error

We are trying to track at least 15 people using these 4 cameras in a 17ft x 32ft room. we were really hoping to use this fusion API for this purpose.

It would be really great if the development team from stereo labs could help to solve this issue.

System configuration:
CPU: 13th Gen Intel® Core™ i9
Memory: 32 GB, 1 x 32 GB, DDR5, 5200 Mhz
GPU: NVIDIA® GeForce RTX™ 4070, 12 GB GDDR6

1 Like