Unusable body tracking with ZED Mini

Software Unity
1

Hello,
I’m working on an experimental project that involves using a group of ZED mini cameras to track the body movements of some subjects, one for each user. The software is developed using Unity, and I’m using the BODY_38_MEDIUM model, CUDA 12.1.1, and ZED SDK version 4.0.7.

Throughout the development phase, I didn’t encounter any issues with the camera or the quality of the tracking. However, recently, as I started setting up for the target users, I’ve encountered significant problems with body tracking using two of the available cameras.
In the first case, it was completely unusable: the avatar instantiated in Unity appeared at random points on the screen and didn’t stay still except for a few frames. I couldn’t stabilize it in any way.
In the second case, initially, the software worked properly, but from a certain point onwards (without making any changes to either the software or hardware), the camera’s video feed started arriving rotated by 180 degrees (occasionally returning to the correct orientation), and the body tracking began behaving as described earlier. Again, I couldn’t revert the situation to normalcy by reconnecting all components or restarting the software or PC.
Both rooms where the cameras were placed were well-lit, and the subjects were properly framed.

Unfortunately, I didn’t have the time to thoroughly analyze the situation in the first case. In the second case, however, I checked in the Depth Viewer, and everything seemed normal. The only thing I noticed was occasional moments when the heatmap briefly returned very different values.
I tried deleting all the used models and re-optimizing them, but nothing changed.
In the first case, I also thought about the possibility of a manufacturing defect, but now with 2 out of 5 cameras showing issues, it seems unlikely (unless it’s an entire problematic batch).

Do you have any ideas on what the problem might be? I’m open to any advice on how to solve this issue or where to investigate further.

Thank you in advance.

2

Hello @lionhead,

  • You’re doing Fusion using Live Link for Unity, is that correct?
  • How are the cameras connected to the PC? Is that over local network or by USB?
  • Do you have the same behaviors on the senders’ side?
  • Do you have the same behavior using a multi-camera SDK sample or is it only in Unity?
  • How much fps are you getting on the sender and receiver side?
  • Can you update to SDK 4.0.8 on the senders and receivers’ side and see if you still have the issue?
3

Hi @JPlou, thak you for your reply.

For my project I have to create a number of individual setups, based on a laptop and a ZED mini (not X mini). Threrefore I’m not using Fusion or Live Link, just the Unity package. The cameras are connected to the laptops using the provided USB cable, orienting the USB-C end properly. The Unity app requires a 1080p 30fps stream and we have no problems with that. Even accessing the camera through the Depth Viewer I can select and receive all the resolution-framerate combinations that the camera supports.

Let me know if this clarifies my usage situation.

4

@lionhead That does clarify the setup!

I’ve got a whole other lot of questions and tests for you though, sorry for that.

  • Does the issue(s) happen directly when you start the app, or over time?
  • Does it happen when using other resolution/fps?
  • Does it happen when the laptop is not on battery/when the laptop power plan is on High Performance?
  • Are you able to reproduce it on another setup? (laptop or not)
  • Are you able to reproduce it in the SDK Body Tracking samples? (They also come in the SDK install folder, under C:\Program Files (x86)\ZED SDK\samples\bin or C:\Program Files (x86)\ZED SDK\samples\body tracking\body tracking )
  • Can you try updating to 4.0.8? There should be no change necessary on the Unity app, just reinstalling the ZED SDK on the PC should suffice.
  • What OS / Version are you running Unity on? Also, which version of Unity?

Also, so that I can investigate on my side if you want:

  • Can you send me an SVO reproducing the issue? It can be recorded directly in Unity through a button in the ZEDManager’s Inspector. You can send it either here or to support@stereolabs.com mentioning this topic, I’ll get it without it being shared here.
5

@JPlou I’m more than happy to provide you additional infos, if they can help in finding the problem.

Does the issue(s) happen directly when you start the app, or over time?

In the first case it started immediately. In the second it started after approximately 15/20 minutes of usage. In both cases the cameras were just unboxed and brand new.

Does it happen when using other resolution/fps?

We did not try yet to use other settings because we must submit to the availability of users. We will try as soon as possible.

Does it happen when the laptop is not on battery/when the laptop power plan is on High Performance?

The laptops are always used connected to the power supply in order to get the best performance, otherwise the game is barely usable. The power play is on high performance, but there is no thermal throttling or general system fatigue. They are all Alienware laptops that we tested before starting the project.

Are you able to reproduce it on another setup? (laptop or not)

At the moment it has only occurred on the two systems I mentioned in the initial post. During the development of the application (on an Alienware desktop workstation) nothing similar has ever occurred

Are you able to reproduce it in the SDK Body Tracking samples?

I will try it asap, like for other resolutions

Can you try updating to 4.0.8? There should be no change necessary on the Unity app, just reinstalling the ZED SDK on the PC should suffice.

We can manage to do so, but since this would imply an update on all setups, I would move this eventuality forward as much as possible and try the other things suggested for the moment

What OS / Version are you running Unity on? Also, which version of Unity?

We are using Windows 11 on all the machines. The app is based on Unity 2022.3.4f1.

Can you send me an SVO reproducing the issue?

Certainly, but not immediately. as soon as one of the interested users becomes available I will do that too and send it to you as soon as possible.

Thank you again for your support and your patience!

6

@lionhead
Thanks for the answers, I’ll wait for the results of your tests.

  • Can you also send me a ZED Diagnostic result from the laptops? (the file you can save at the end of ZED Diagnostic)

  • A quick test could also be to try re-optimizing the AI models. (Even if it should not impact your results as much as the issue you’re facing right now).

7

I already deleted and optimized from scratch a couple times all the models involved. It was one of the first attempts to solve the problem.
Also, the ZED Diagnostic checks are all passed, except for the one on the AI models which points out that not all the models have been optimized, but it’s fine, we optimized only the ones that we need for the project. In any case, I’ll send you the generated report file.

Tomorrow I’ll be able to work on one of the two problematic setups. I’ll update you during late afternoon or on friday morning.

1 Like
8

Hi @JPlou,
I managed to do the tests on the camera, as I promised. I just sent you the files that you required me via email.

in response to the remaining open questions:

  • I built the app using different resolution and/or fps and/or depth computation model and nothing changed
  • I used the body tracking C++ sample together with the BODY_38_ACCURATE AI model and the instantiated skeletons behaved in a similar way to what was found in our application.
  • The ZED diagnostic report doesn’t seem to highlight any particular problems (but check it yourself because you are surely more experienced)
  • The recorded SVO looks fine

Triggered by the fact that the camera seemed unable to detect the correct orientation, I checked also the sensors’ data visualized by ZED Sensor Viewer. There I immediately noticed that the visualized camera model continuously flipped up and down and this behaviour was justified by a decidedly anomalous rotation speed on the X axis, considering that the camera is used stationary on a tripod. To show you this behaviour I sent both a csv sensor data record and a screen capture of the ZED Sensor Viewer plots.
I don’t know how much this problem affects the overall outcome, but it’s definitely not regular. In the video I sent in which I show the problem contextualized in our application, although not isolated on a single axis, a certain rotationality of the defect is perceived in a certain sense, so perhaps things are connected.

Take your time to view and analyze the material. For any other clarification or test that may seem useful to you, please ask and I will try to do it.
Tomorrow I will have access to the other problematic camera and I will let you know if the same problem is also present on it.

1 Like
10

Hi @JPlou,
as anticipated I managed to test again the other camera too.
Differently from the first usage and from the other camera, this time the body tracking was not perfect, but at least usable. The avatar was not scattered around the screen.
However, we encountered a couple of app freezes. After them, restarting the app, we noticed a constant sensible rotation of the avatar (approximately 40/45 degrees to the right) even if the person was standing in front of the camera facing it perfectly. After a few minutes in this condition, the camera turned off on its own and we needed to disconnect and reconnect the USB cable in order to use it again.

We run the body tracking sample app at the beginning and it worked fine.
After the last reconnection I launched ZED Sensors Viewer and in my impression there was nothing strange.
I attach to this post the csv file containing approximately 10 seconds of sensor data.
20240301_170314-ZED-M-sn_10543771-ZEDSensorsData.csv (686.6 KB)

I hope that this further information, although not very aligned with that of the other camera, can be of help to you.
Let me know if you discover anything or need anything else.

11

Thank you for the files and additional report @lionhead, I will take a look as soon as possible.

Jean-Loup MACARIT
R&D Engineer
Stereolabs Support

12

Hello @lionhead,

First of all, I’m so sorry for the delay.

It seems like an IMU issue.
You can restore the IMU calibration running the “ZED Calibration” tool from the command line, adding the flag --cimu or --calib_imu.
That would be, open a command line in the tools folder of the SDK installation, and run

'.\ZED Calibration.exe' --cimu

Before running the calibration process please be sure that the camera is completely steady and placed on a horizontal surface.
The re-calibration process is fast and fully automated, you can test if it worked correctly using the ZED Sensor Viewer tool.

Please let me know if it helps. You could run it on your second camera too to try and fix the drift.