So far, this part works well: the client runs at a stable 30 FPS. We have also tested single-camera skeleton detection independently and found the results to be of adequate quality.
Each Jetson is connected via a private network to a third machine (Intel i9 + RTX 4090, running Windows and ZED SDK 5.0.7).
We then start ZED360 and configure it using the Jetsons’ IP addresses and the ZED camera serial numbers. However, to our surprise, the incoming streams fluctuate between 10 and 15 FPS.
We observe similar results when running ZED360 on a third Jetson.
We ran an additional test by writing custom code that receives both camera streams via the Fusion API. Once again, the observed FPS is significantly lower than expected and fluctuates between 15 and 25 FPS, making the setup unusable.
Based on these tests, we believe that the Fusion SDK is receiving the skeleton streams at a lower frame rate than the 30 FPS at which they are being sent.
We would appreciate your assistance in identifying the failure point and understanding why the streams are not being received at the expected frame rate, especially since we do not have access to the Fusion SDK internals.
Please let us know if you need additional details or access to the specific code we are using.
All the components of the rig are indded synchronised by PTP. The slave is reporting an offset in the tens of thousands of nanoseconds, which I’m assmuming is sufficient.
Hi @Myzhar . I have upgraded to 5.1.2, but but now ZED360 has stopped working and is returning an error: ERROR: WRONG BODY FORMAT. No sekeleton are showing in ZED360. I have tried all 3 formats on the sender end (18, 34, 38), same error. What format is ZED360 expecting?
Here’s my sender code:
// ZED includes
#include <sl/Camera.hpp>
// Using std and sl namespaces
using namespace std;
using namespace sl;
void print(string msg_prefix, ERROR_CODE err_code = ERROR_CODE::SUCCESS, string msg_suffix = "");
void parseArgs(int argc, char **argv, InitParameters& param);
int main(int argc, char **argv) {
// Create ZED Bodies
Camera zed;
InitParameters init_parameters;
init_parameters.camera_resolution = RESOLUTION::AUTO;
init_parameters.depth_mode = DEPTH_MODE::NEURAL_LIGHT;
init_parameters.coordinate_system = COORDINATE_SYSTEM::RIGHT_HANDED_Y_UP;
// Open the camera
auto returned_state = zed.open(init_parameters);
if (returned_state != ERROR_CODE::SUCCESS) {
zed.close();
return EXIT_FAILURE;
}
// Enable Positional tracking (mandatory for object detection)
PositionalTrackingParameters positional_tracking_parameters;
//If the camera is static, uncomment the following line to have better performances
positional_tracking_parameters.set_as_static = true;
returned_state = zed.enablePositionalTracking(positional_tracking_parameters);
if (returned_state != ERROR_CODE::SUCCESS) {
zed.close();
return EXIT_FAILURE;
}
// Enable the Body tracking module
BodyTrackingParameters body_tracker_params;
body_tracker_params.enable_tracking = false; // track people across images flow
body_tracker_params.enable_body_fitting = false; // smooth skeletons moves
body_tracker_params.body_format = sl::BODY_FORMAT::BODY_18;
body_tracker_params.detection_model = BODY_TRACKING_MODEL::HUMAN_BODY_FAST;
//body_tracker_params.allow_reduced_precision_inference = true;
returned_state = zed.enableBodyTracking(body_tracker_params);
if (returned_state != ERROR_CODE::SUCCESS) {
zed.close();
return EXIT_FAILURE;
}
// Configure object detection runtime parameters
BodyTrackingRuntimeParameters body_tracker_parameters_rt;
body_tracker_parameters_rt.detection_confidence_threshold = 40;
body_tracker_parameters_rt.skeleton_smoothing = 0.7;
CommunicationParameters configuration;
configuration.setForLocalNetwork(30000);
zed.startPublishing(configuration);
// Create ZED Bodies filled in the main loop
Bodies bodies;
while (true) {
// Grab images
auto err = zed.grab();
if (err == ERROR_CODE::SUCCESS) {
// Retrieve Detected Human Bodies
zed.retrieveBodies(bodies, body_tracker_parameters_rt);
}
}
// Release Bodies
bodies.body_list.clear();
// Disable modules
zed.disableBodyTracking();
zed.disablePositionalTracking();
zed.close();
return EXIT_SUCCESS;
}
Hi, this means the fusion module is receiving data from different formats coming from each camera.
Can you double-check each sender is using the same SDK version and the same body format as well?
I double checked and both senders and receiver are using the same SDK version (5.1.2) and the sender code I posted above (so sl::BODY_FORMAT::BODY_18).
Some additional context:
The error happens consistently and no skeletons appear. However once in a while (1 out 10, or 1 out of 20). There’s no error, the skeletons are displayed, and everything works impeccably. I was not able to isolate the factor that causes it to work, seems to be random.
This error started appearing when I upgraded from 5.0.7 to 5.1.2, it wasn’t present before. I upgraded because 5.0.7 had another issue (low framerate, which is now solved).
Both ZED Live Link and my custom receiver work consistly fine. It’s only ZED360 which is causing issues. Therefore, I have no way of debugging this, as the calibration functions are not exposed in the API afaik.
It’s quite frustrating because this look like the last hurdle before having a functional system, everything else seems to be working fine. Your help would be much appreciated. Please let me know if there’s any useful tests I can run or if you have any sender code that’s proven to work with this SDK version.
It looks very similar to what you’re using, so I don’t think the issue is coming from the sender side.
Are the Jetsons connected over Wi-Fi? If so, this could explain the inconsistent behavior. For this type of Fusion network workflow, we strongly recommend using a wired Ethernet connection to ensure stable bandwidth.
I’m Elsa, Senior Producer working with Shadi at FIELD.IO. This matter is now halting our production timeline, with a big Korean automotive client waiting and pause on further large-scale purchases from StereoLab.
Can we please escalate this into a phone-call or a meeting, either today or by Monday morning latest?
We understand the urgency of your situation. Our team is actively investigating the possible causes of the issue described by your colleagues.
At the moment, we don’t yet have a definitive update to share, which is why your tickets haven’t been updated. Please be assured that the lack of visible replies does not mean the issue is being ignored.
Customer satisfaction remains a top priority for us, and we are doing our best to support you and all our customers as quickly and thoroughly as possible.
Thank you for your reply, it’s highly appreciated.
We will need to make a decision by Monday whether or not to move forward with StereoLab or another company for this project and therefore time is of the essence.
Could we please set up a meeting to resolve this together? I believe this would be the most efficient use of time for everyone involved. This way we can easier get to the bottom of the issue and have faster communication between our developers and yours.
Please let me know your availability for this as soon as possible.
We still haven’t received an update on the issue in your ZED360 calibration software.
As mentioned last week, we need to make a decision now on whether or not to go forward with StereoLabs on this project. Could you please share with us the resolve status of this issue? Are we correct in thinking this is a bug on your end, in which case, when can it be fixed?
I can push the decision-making of which system/company to go with until end of this weeklatest—but in that case, we need to have some reassurance that this can be resolved.
Please consider giving me a call to discuss this, especially given your top priority on customer satisfaction,
I really like the zed2i cameras. But the fusion network workflow isn’t reliable for customer projects and software support is catastrophic (i guess too many construction sites). We also had a big costumer project, which we could only rescue buying old 40xx graphics cards and using 4.x sdk. But this isn’t a workaround for future projects. So we are also looking for a different solution for future projects.
I’m sure the developers and support team is doing there best… but it don’t work out. But Stereolabs should focus on core features and not doing false promises…
Same issue over here. Stock Jetson ZEDbox running iirc 5.0.7. Two individual camera grabbers using the provided sample code completely saturate the box’s wattage in MAXN mode, but with tuned settings, we can get both reporting ballpark 30fps grab. Any Fusion receiver build we connect to that dual IP data stream does not run more than ballpark 15fps. We have been given the runaround about using PTP to sync the sender and receivers, but not only is this very nontrivial when the receiver is Windows, it is mooted by the fact that a Fusion receiver running on the same box gets identical 15fps performance to a Fusion receiver running remotely, without meaningfully impacting the sender grab rate.
Right now, we are experimenting with running the full Fusion stack on the Jetson, which gets us ~24fps, but we still have an independent issue of bodies changing IDs when they cross from camera to camera, and the additional issue that the Fusion project provided as a sample does not seem to expose its data port (20001) to external machines, and the whole network stack underlying Fusion looks to be API-obscured. About the best bodge we can make at present is to run not only the full camera-to-Fusion stack on the Jetson, but also an additional layer of bespoke middleware to consume the raw UDP 20001 unicast data and relay it on a more flexible TCP socket for external use.
(Completely independently, we are also in the position of needing to maintain long-term 4.x projects which are no longer compatible with available (read: RTX 5xxx) GPUs, but that deserves another thread entirely)
As an update, with the same hardware (ZEDBOX MINI), running the same full-stack 2x GMSL2 ZED X MINI Fusion end-to-end locally with NEURAL_LIGHT depth mode and HUMAN_BODY_FAST detection HD1080 30fps requested,
SDK 4.2.5 flat out cannot find the cameras, even with latest drivers, and a support email response indicates this is expected. (but a full Windows 4.x stack on a RTX 4080 does not have any issues with framerate or body ID)
SDK 5.0.7 clocks 24fps fused, dropping to 20-22 with a user in-scene, but can’t maintain body ID across cameras. When using individual grabbers sending over IP, this drops to 15fps fused whether the receiver is local or remote.
SDK 5.1.2 clocks 15fps fused, weirdly rising to 18fps with a user being tracked, but still can’t maintain body ID.
A support email response indicates `There are no slowing down verified issues with Fusion and Body Tracking. The SDK team was not able to replicate them.`