So far, this part works well: the client runs at a stable 30 FPS. We have also tested single-camera skeleton detection independently and found the results to be of adequate quality.
Each Jetson is connected via a private network to a third machine (Intel i9 + RTX 4090, running Windows and ZED SDK 5.0.7).
We then start ZED360 and configure it using the Jetsons’ IP addresses and the ZED camera serial numbers. However, to our surprise, the incoming streams fluctuate between 10 and 15 FPS.
We observe similar results when running ZED360 on a third Jetson.
We ran an additional test by writing custom code that receives both camera streams via the Fusion API. Once again, the observed FPS is significantly lower than expected and fluctuates between 15 and 25 FPS, making the setup unusable.
Based on these tests, we believe that the Fusion SDK is receiving the skeleton streams at a lower frame rate than the 30 FPS at which they are being sent.
We would appreciate your assistance in identifying the failure point and understanding why the streams are not being received at the expected frame rate, especially since we do not have access to the Fusion SDK internals.
Please let us know if you need additional details or access to the specific code we are using.
All the components of the rig are indded synchronised by PTP. The slave is reporting an offset in the tens of thousands of nanoseconds, which I’m assmuming is sufficient.
Hi @Myzhar . I have upgraded to 5.1.2, but but now ZED360 has stopped working and is returning an error: ERROR: WRONG BODY FORMAT. No sekeleton are showing in ZED360. I have tried all 3 formats on the sender end (18, 34, 38), same error. What format is ZED360 expecting?
Here’s my sender code:
// ZED includes
#include <sl/Camera.hpp>
// Using std and sl namespaces
using namespace std;
using namespace sl;
void print(string msg_prefix, ERROR_CODE err_code = ERROR_CODE::SUCCESS, string msg_suffix = "");
void parseArgs(int argc, char **argv, InitParameters& param);
int main(int argc, char **argv) {
// Create ZED Bodies
Camera zed;
InitParameters init_parameters;
init_parameters.camera_resolution = RESOLUTION::AUTO;
init_parameters.depth_mode = DEPTH_MODE::NEURAL_LIGHT;
init_parameters.coordinate_system = COORDINATE_SYSTEM::RIGHT_HANDED_Y_UP;
// Open the camera
auto returned_state = zed.open(init_parameters);
if (returned_state != ERROR_CODE::SUCCESS) {
zed.close();
return EXIT_FAILURE;
}
// Enable Positional tracking (mandatory for object detection)
PositionalTrackingParameters positional_tracking_parameters;
//If the camera is static, uncomment the following line to have better performances
positional_tracking_parameters.set_as_static = true;
returned_state = zed.enablePositionalTracking(positional_tracking_parameters);
if (returned_state != ERROR_CODE::SUCCESS) {
zed.close();
return EXIT_FAILURE;
}
// Enable the Body tracking module
BodyTrackingParameters body_tracker_params;
body_tracker_params.enable_tracking = false; // track people across images flow
body_tracker_params.enable_body_fitting = false; // smooth skeletons moves
body_tracker_params.body_format = sl::BODY_FORMAT::BODY_18;
body_tracker_params.detection_model = BODY_TRACKING_MODEL::HUMAN_BODY_FAST;
//body_tracker_params.allow_reduced_precision_inference = true;
returned_state = zed.enableBodyTracking(body_tracker_params);
if (returned_state != ERROR_CODE::SUCCESS) {
zed.close();
return EXIT_FAILURE;
}
// Configure object detection runtime parameters
BodyTrackingRuntimeParameters body_tracker_parameters_rt;
body_tracker_parameters_rt.detection_confidence_threshold = 40;
body_tracker_parameters_rt.skeleton_smoothing = 0.7;
CommunicationParameters configuration;
configuration.setForLocalNetwork(30000);
zed.startPublishing(configuration);
// Create ZED Bodies filled in the main loop
Bodies bodies;
while (true) {
// Grab images
auto err = zed.grab();
if (err == ERROR_CODE::SUCCESS) {
// Retrieve Detected Human Bodies
zed.retrieveBodies(bodies, body_tracker_parameters_rt);
}
}
// Release Bodies
bodies.body_list.clear();
// Disable modules
zed.disableBodyTracking();
zed.disablePositionalTracking();
zed.close();
return EXIT_SUCCESS;
}
Hi, this means the fusion module is receiving data from different formats coming from each camera.
Can you double-check each sender is using the same SDK version and the same body format as well?
I double checked and both senders and receiver are using the same SDK version (5.1.2) and the sender code I posted above (so sl::BODY_FORMAT::BODY_18).
Some additional context:
The error happens consistently and no skeletons appear. However once in a while (1 out 10, or 1 out of 20). There’s no error, the skeletons are displayed, and everything works impeccably. I was not able to isolate the factor that causes it to work, seems to be random.
This error started appearing when I upgraded from 5.0.7 to 5.1.2, it wasn’t present before. I upgraded because 5.0.7 had another issue (low framerate, which is now solved).
Both ZED Live Link and my custom receiver work consistly fine. It’s only ZED360 which is causing issues. Therefore, I have no way of debugging this, as the calibration functions are not exposed in the API afaik.
It’s quite frustrating because this look like the last hurdle before having a functional system, everything else seems to be working fine. Your help would be much appreciated. Please let me know if there’s any useful tests I can run or if you have any sender code that’s proven to work with this SDK version.
It looks very similar to what you’re using, so I don’t think the issue is coming from the sender side.
Are the Jetsons connected over Wi-Fi? If so, this could explain the inconsistent behavior. For this type of Fusion network workflow, we strongly recommend using a wired Ethernet connection to ensure stable bandwidth.
I’m Elsa, Senior Producer working with Shadi at FIELD.IO. This matter is now halting our production timeline, with a big Korean automotive client waiting and pause on further large-scale purchases from StereoLab.
Can we please escalate this into a phone-call or a meeting, either today or by Monday morning latest?
We understand the urgency of your situation. Our team is actively investigating the possible causes of the issue described by your colleagues.
At the moment, we don’t yet have a definitive update to share, which is why your tickets haven’t been updated. Please be assured that the lack of visible replies does not mean the issue is being ignored.
Customer satisfaction remains a top priority for us, and we are doing our best to support you and all our customers as quickly and thoroughly as possible.
Thank you for your reply, it’s highly appreciated.
We will need to make a decision by Monday whether or not to move forward with StereoLab or another company for this project and therefore time is of the essence.
Could we please set up a meeting to resolve this together? I believe this would be the most efficient use of time for everyone involved. This way we can easier get to the bottom of the issue and have faster communication between our developers and yours.
Please let me know your availability for this as soon as possible.