Framerate issue when using the Fusion Network Workflow

Software
,
1

Hello,

We are currently experiencing issues calibrating a multi-camera ZED 2i setup using the Fusion Network workflow.

We are experimenting with a two-camera configuration, each camera connected to a Jetson Orin Nano 8GB running ZED SDK 5.0.7.

We have built a simple sender client based on the sample code provided here: https://www.stereolabs.com/docs/fusion/overview

So far, this part works well: the client runs at a stable 30 FPS. We have also tested single-camera skeleton detection independently and found the results to be of adequate quality.

Each Jetson is connected via a private network to a third machine (Intel i9 + RTX 4090, running Windows and ZED SDK 5.0.7).

We then start ZED360 and configure it using the Jetsons’ IP addresses and the ZED camera serial numbers. However, to our surprise, the incoming streams fluctuate between 10 and 15 FPS.

We observe similar results when running ZED360 on a third Jetson.

We ran an additional test by writing custom code that receives both camera streams via the Fusion API. Once again, the observed FPS is significantly lower than expected and fluctuates between 15 and 25 FPS, making the setup unusable.

Based on these tests, we believe that the Fusion SDK is receiving the skeleton streams at a lower frame rate than the 30 FPS at which they are being sent.

We would appreciate your assistance in identifying the failure point and understanding why the streams are not being received at the expected frame rate, especially since we do not have access to the Fusion SDK internals.

Please let us know if you need additional details or access to the specific code we are using.

Thanks in advance,

Shadi

2

Hi @shadi
Welcome to the Stereolabs community.

Please verify that the system time of all the components of the rig is synchronized.

You could use PTP (or any other method) to guarantee it.

3

Thanks for getting back to me, @Myzhar.

All the components of the rig are indded synchronised by PTP. The slave is reporting an offset in the tens of thousands of nanoseconds, which I’m assmuming is sufficient.

[122.322] master offset -11530 s2 freq -22665 path delay 93340
[123.322] master offset -12272 s2 freq -22751 path delay 94017
[124.322] master offset -19051 s2 freq -23448 path delay 97486
[125.322] master offset -13736 s2 freq -22930 path delay 94313
[126.322] master offset -6967 s2 freq -22260 path delay 93917
[127.322] master offset -7275 s2 freq -22298 path delay 93917
[128.323] master offset 8293 s2 freq -20733 path delay 93917
[129.323] master offset -7109 s2 freq -22281 path delay 95601
[130.323] master offset -30849 s2 freq -24685 path delay 95601
[131.323] master offset -8630 s2 freq -22472 path delay 94618

4

Before proceeding further, I recommend you upgrade the ZED SDK on all the machines to the latest v5.1.2: https://www.stereolabs.com/developers/release

5

(post deleted by author)

6

Hi @Myzhar . I have upgraded to 5.1.2, but but now ZED360 has stopped working and is returning an error: ERROR: WRONG BODY FORMAT. No sekeleton are showing in ZED360. I have tried all 3 formats on the sender end (18, 34, 38), same error. What format is ZED360 expecting?

Here’s my sender code:

// ZED includes
#include <sl/Camera.hpp>


// Using std and sl namespaces
using namespace std;
using namespace sl;

void print(string msg_prefix, ERROR_CODE err_code = ERROR_CODE::SUCCESS, string msg_suffix = "");
void parseArgs(int argc, char **argv, InitParameters& param);

int main(int argc, char **argv) {

    // Create ZED Bodies
    Camera zed;
    InitParameters init_parameters;
    init_parameters.camera_resolution = RESOLUTION::AUTO;
    init_parameters.depth_mode = DEPTH_MODE::NEURAL_LIGHT;
    init_parameters.coordinate_system = COORDINATE_SYSTEM::RIGHT_HANDED_Y_UP;

    // Open the camera
    auto returned_state = zed.open(init_parameters);
    if (returned_state != ERROR_CODE::SUCCESS) {
        zed.close();
        return EXIT_FAILURE;
    }

    // Enable Positional tracking (mandatory for object detection)
    PositionalTrackingParameters positional_tracking_parameters;
    //If the camera is static, uncomment the following line to have better performances
    positional_tracking_parameters.set_as_static = true;

    returned_state = zed.enablePositionalTracking(positional_tracking_parameters);
    if (returned_state != ERROR_CODE::SUCCESS) {
        zed.close();
        return EXIT_FAILURE;
    }

    // Enable the Body tracking module
    BodyTrackingParameters body_tracker_params;
    body_tracker_params.enable_tracking = false; // track people across images flow
    body_tracker_params.enable_body_fitting = false; // smooth skeletons moves
    body_tracker_params.body_format = sl::BODY_FORMAT::BODY_18;
    body_tracker_params.detection_model = BODY_TRACKING_MODEL::HUMAN_BODY_FAST;
    //body_tracker_params.allow_reduced_precision_inference = true;

    returned_state = zed.enableBodyTracking(body_tracker_params);
    if (returned_state != ERROR_CODE::SUCCESS) {
        zed.close();
        return EXIT_FAILURE;
    }

    // Configure object detection runtime parameters
    BodyTrackingRuntimeParameters body_tracker_parameters_rt;
    body_tracker_parameters_rt.detection_confidence_threshold = 40;
    body_tracker_parameters_rt.skeleton_smoothing = 0.7;
    
    CommunicationParameters configuration;
    configuration.setForLocalNetwork(30000);
    zed.startPublishing(configuration);

    // Create ZED Bodies filled in the main loop
    Bodies bodies;
    while (true) {
        // Grab images
        auto err = zed.grab();
        if (err == ERROR_CODE::SUCCESS) {
            // Retrieve Detected Human Bodies
            zed.retrieveBodies(bodies, body_tracker_parameters_rt);
        } 
    }

    // Release Bodies
    bodies.body_list.clear();

    // Disable modules
    zed.disableBodyTracking();
    zed.disablePositionalTracking();
    zed.close();

    return EXIT_SUCCESS;
}