Questions regarding multiple cameras fusion API

[felixshing]

I would like to ask several questions regarding the new camera fusion API

1. I plan to use 3 or 4 cameras to capture human body data. Can this API help me fusion the generated point cloud, mesh, and 3D keypoints for the human body?
2. If I plan to attach multiple cameras to one server, how many zed cameras I can use at the same time? Is the 3080 GPU enough to handle 3 or 4 Zed 2i cameras?
3. It seems that BODY70 API is removed in the new zed SDK. If I choose to use another project, such as MinimalHand (CVPR 2020), to extract hand keypoints, it will only give us 2D keypoints. How I can leverage depth information to lift the extracted 2D keypoints to 3D?
4. This question is about the whole ZED SDK. Regarding the running speed, will CPP version faster then Python version?

[Myzhar]

Hi @felixshing

@felixshing

I plan to use 3 or 4 cameras to capture human body data. Can this API help me fusion the generated point cloud, mesh, and 3D keypoints for the human body?

You can actually retrieve only the fused information regarding the skeleton key points

@felixshing

If I plan to attach multiple cameras to one server, how many zed cameras I can use at the same time? Is the 3080 GPU enough to handle 3 or 4 Zed 2i cameras?

The number of cameras scales with the power of the data-collecting CPU. More power and more RAM means more camera data inputs.

@felixshing

It seems that BODY70 API is removed in the new zed SDK. If I choose to use another project, such as MinimalHand (CVPR 2020), to extract hand keypoints, it will only give us 2D keypoints. How I can leverage depth information to lift the extracted 2D keypoints to 3D?

The 3D information and the 2D information are always strictly related with math formulas:

How can I convert 3D world coordinates to 2D image coordinates and viceversa?

@felixshing

This question is about the whole ZED SDK. Regarding the running speed, will CPP version faster then Python version?

The ZED SDK is natively written in C++ language, so you can expect it to be faster.

PS You can find long discussions online about who is faster between Python and C++. I prefer not to start a new one in this context

[felixshing]

Thank you for your reply. Regarding the fusion of skeleton keypoints, if I use other keypoint detection algorithms, such as Google’s MediaPipe and MinimalHand, instead of Zed’s SDK, can Zed fusion API still be able to fuse the data? Moreover, I would like to ask when will the BODY70 API come back?

Another question is that, if I have multiple jetson AI boards (such as Jetson NX), I should be able to conduct “network” fusion setup described in Fusion | Stereolabs. Is it correct? In such case, does the connection between jetson and the server necessarily need to be cable? is it feasible to connect in a WiFi environment?

[Myzhar]

@felixshing

if I use other keypoint detection algorithms, such as Google’s MediaPipe and MinimalHand, instead of Zed’s SDK, can Zed fusion API still be able to fuse the data?

No, you must merge external data with your own algorithms.

@felixshing

Moreover, I would like to ask when will the BODY70 API come back?

We are evaluating it. When (if) we reach a good detection stability we will re-enable it.

@felixshing

In such case, does the connection between jetson and the server necessarily need to be cable? is it feasible to connect in a WiFi environment?

You can use WiFi, but the connection must be reliable and the streaming correctly tuned to fit the bandwidth of your connection.

[felixshing]

Thank you for your reply. Regarding fusing keypoints got from other algorithms, could you give me some hints that how to implement it, for example, some papers or open-source algorithms?