ZED SDK 5.3.0: SVO2 recording FPS drops over time when body tracking is enabled

Software ROS2
,
1

ZED SDK 5.3.0: SVO2 recording FPS drops over time when body tracking is enabled on ZED X + AGX Orin, while SDK 5.1.1 works with ZED_SDK_LIFT_ASYNC_IMAGE_RETRIEVAL_RESTRICTION=1

log
log1920×1680 686 KB

Environment

  • Camera: ZED X
  • Platform: NVIDIA Jetson AGX Orin
  • Power mode: 50W
  • ZED SDK version tested:
    • 5.3.0: issue reproduced
    • 5.1.1: appears to work correctly
  • Recording format: SVO2
  • Resolution/FPS: HD1080 @ 60 FPS
  • Depth mode: NEURAL_LIGHT
  • Body tracking:
    • BODY_FORMAT::BODY_18
    • BODY_TRACKING_MODEL::HUMAN_BODY_FAST
    • body fitting disabled
  • Recording compression: H265
  • Runtime environment:
    • sudo jetson_clocks enabled
    • ZED_SDK_LIFT_ASYNC_IMAGE_RETRIEVAL_RESTRICTION=1 exported before running

Issue Summary

I am trying to record 1080p60 SVO2 data from a ZED X camera on an AGX Orin while also running online body tracking.

With ZED SDK 5.3.0, recording starts close to 60 FPS, but after some time the recording FPS starts dropping and becomes unstable. The drop happens even when body tracking is throttled to 20 FPS and ROS publishing is only done when a new body tracking result is available.

In contrast, with ZED SDK 5.1.1, setting:

export ZED_SDK_LIFT_ASYNC_IMAGE_RETRIEVAL_RESTRICTION=1

appears to keep SVO2 recording stable at 60 FPS while body tracking is enabled.

This makes it look like the async image retrieval / recording isolation behavior changed or regressed between SDK 5.1.1 and SDK 5.3.0.

Expected Behavior

When async_image_retrieval = false and:

ZED_SDK_LIFT_ASYNC_IMAGE_RETRIEVAL_RESTRICTION=1

is set, I expect SVO2 recording to stay at the camera FPS, i.e. around 60 FPS, even if body tracking processing is slower or throttled.

In other words:

  • SVO2 recording should remain close to 60 FPS
  • Body tracking may run at a lower rate, for example 20 FPS
  • Body tracking should not drag down the recording FPS over time

Actual Behavior with SDK 5.3.0

With SDK 5.3.0:

  • Recording starts near 60 FPS
  • grab FPS also starts near 60 FPS
  • body FPS is around 20 FPS after throttling
  • After some time, both recording and grab FPS become unstable and drop
  • rec ingested FPS and rec encoded FPS drop together

Example behavior from logs:

rec ingested FPS: 60 | rec encoded FPS: 60 | grab FPS: 60 | body FPS: 20

at the beginning, then later:

rec ingested FPS: 40~50 | rec encoded FPS: 40~50 | grab FPS: 40~50 | body FPS: 10~17

and sometimes even lower.

The important point is that:

rec ingested FPS ~= rec encoded FPS

So the bottleneck does not seem to be H265 encoding or disk writing. The slowdown appears to happen before encoding, around grab() / SDK internal processing.

Minimal Pipeline

The application does the following:

sl::InitParameters init_params;
init_params.camera_resolution = sl::RESOLUTION::HD1080;
init_params.camera_fps = 60;
init_params.coordinate_units = sl::UNIT::METER;
init_params.coordinate_system = sl::COORDINATE_SYSTEM::RIGHT_HANDED_Z_UP_X_FWD;
init_params.depth_mode = sl::DEPTH_MODE::NEURAL_LIGHT;
init_params.async_image_retrieval = false;
init_params.sdk_verbose = 1;

zed.open(init_params);

Then:

sl::PositionalTrackingParameters tracking_params;
zed.enablePositionalTracking(tracking_params);

Then:

sl::BodyTrackingParameters body_params;
body_params.enable_tracking = true;
body_params.enable_body_fitting = false;
body_params.body_format = sl::BODY_FORMAT::BODY_18;
body_params.detection_model = sl::BODY_TRACKING_MODEL::HUMAN_BODY_FAST;

zed.enableBodyTracking(body_params);

Then recording:

sl::RecordingParameters rec_params;
rec_params.video_filename.set("recording.svo2");
rec_params.compression_mode = sl::SVO_COMPRESSION_MODE::H265;

zed.enableRecording(rec_params);

The main loop keeps grab() running as fast as possible:

while (running) {
    auto err = zed.grab(runtime_params);

    if (err == sl::ERROR_CODE::SUCCESS) {
        // SVO recording is expected to follow camera FPS

        // Body tracking is throttled to 20 FPS:
        if (frame_counter >= 3) {
            frame_counter = 0;
            sl::Bodies bodies;
            zed.retrieveBodies(bodies, body_rt);
        }
    }
}

runtime_params.enable_depth = true.

Tests Already Performed

1. Pure SVO2 recording only

A separate recording-only program using the same ZED X camera and 1080p60 mode records SVO2 perfectly at 60 FPS.

This suggests:

  • Camera mode is correct
  • SVO2 recording alone is fine
  • H265 encoding alone is fine
  • Disk writing is likely not the primary bottleneck

2. Recording + body tracking with SDK 5.3.0

When body tracking is enabled in the same application, recording FPS starts near 60 but drops over time.

3. Body tracking throttled to 30 FPS and then 20 FPS

I throttled retrieveBodies():

  • 30 FPS: better than running every frame, but still unstable over time
  • 20 FPS: more stable, but still eventually drops and becomes unstable

This suggests body tracking load contributes to the issue, but throttling does not fully isolate recording.

4. ROS publishing removed as a bottleneck

Originally body data was published in a separate ROS2 thread at 30 FPS. I changed it so publishing only happens when a new body tracking result is available.

The issue still happens, so ROS publishing is not the primary cause.

5. tegrastats monitoring

With sudo jetson_clocks enabled:

  • CPU/GPU temperatures remain low, around 50C
  • No obvious thermal throttling
  • CPU clocks are locked
  • GPU/GR3D and NVENC are active but not always saturated
  • rec ingested FPS and rec encoded FPS drop together

This again suggests the slowdown happens before encoding, likely inside the SDK capture / processing pipeline.

6. Periodic body tracking reset experiment

I also tested periodically disabling and re-enabling body tracking every 60 seconds.

Result:

  • It gives only a short and partial recovery
  • It does not restore the initial stable 60 FPS behavior
  • Later resets become less effective

This suggests the issue is not only inside the body tracking module itself, but possibly in a larger SDK pipeline / buffering / synchronization path.

Comparison with SDK 5.1.1

With ZED SDK 5.1.1 and:

export ZED_SDK_LIFT_ASYNC_IMAGE_RETRIEVAL_RESTRICTION=1

the same type of setup appears to keep SVO2 recording stable at 60 FPS while body tracking is enabled.

With ZED SDK 5.3.0, the same environment variable improves initial behavior but does not prevent recording FPS from eventually dropping.

This makes me suspect that the behavior of:

ZED_SDK_LIFT_ASYNC_IMAGE_RETRIEVAL_RESTRICTION=1

or the async image retrieval / recording pipeline changed between 5.1.1 and 5.3.0.

Questions

  1. Is ZED_SDK_LIFT_ASYNC_IMAGE_RETRIEVAL_RESTRICTION=1 still supported in ZED SDK 5.3.x?
  2. Is there a replacement setting in SDK 5.3.x to ensure SVO2 recording remains at camera FPS while AI modules such as body tracking run slower?
  3. Is this a known regression or behavior change between SDK 5.1.1 and 5.3.0?
  4. What is the recommended way to record 1080p60 SVO2 while simultaneously running body tracking on AGX Orin?
  5. Should recording and body tracking be expected to be fully decoupled when async_image_retrieval = false is used?
  6. Are there recommended body tracking / recording parameters for ZED X on AGX Orin to maintain SVO2 at 60 FPS?

Current Workaround

The most stable workaround found so far is:

  • Use sudo jetson_clocks
  • Export ZED_SDK_LIFT_ASYNC_IMAGE_RETRIEVAL_RESTRICTION=1
  • Use async_image_retrieval = false
  • Throttle body tracking to 20 FPS
  • Publish body data only when retrieveBodies() returns new data

But even with this, SDK 5.3.0 still eventually shows recording FPS instability.

The more reliable workaround appears to be:

  • Use SDK 5.1.1 for online SVO2 recording
  • Use SDK 5.3.0 offline for body tracking / gait analysis from the recorded SVO2 file

Additional Notes

In the logs, I added separate counters for:

rec ingested FPS
rec encoded FPS
grab FPS
body FPS

The most important observation is that rec ingested FPS and rec encoded FPS drop together, meaning the encoder is not falling behind. Instead, the recording input rate itself is reduced. This points to upstream SDK pipeline slowdown rather than disk or encoder throughput.

2

We do not recommend recording SVO files while a high-demanding module, such as Body Tracking, is enabled.

This is because the Jetson module is probably throttled. Have you checked the temperature?

This is expected.

I will report this to the ZED SDK team to verify it and eventually fix it.