Subject: 62% frame drops with auto-exposure enabled on 4x ZED X (GMSL2) — data attached
Hi Stereolabs team,
We’re running 4x ZED X Mini (ZED-XG 120mm) cameras over GMSL2 on a Jetson AGX Orin 64GB (JetPack 6.2.1 / L4T 36.4.4) with ZED SDK 5.2. We capture 1080p30 from all 4 stereo pairs simultaneously (8 video streams) using a custom gRPC capture service with NVENC hardware encoding.
The issue: When cameras are in motion with auto-exposure enabled, grab() drops ~62% of frames. The problem disappears completely when exposure is locked to manual mode.
Test Results
We ran two back-to-back recordings with the same walking path through a house (mixed lighting — bright windows, dim hallways):
| Metric | Auto-exposure ON (32s) | Exposure locked (19s) |
|—|—|—|
| Frame drop rate | 62–63% across all cameras | 0.0% across all cameras |
| Effective frame rate | 11.3 Hz (38% of expected 30 Hz) | 30.0 Hz (100% of expected) |
| Frames captured | 354–366 per camera (of ~966 expected) | 577–578 per camera (of ~577 expected) |
| Full sync groups (all 8 streams) | 137 of 724 (19%) | 576 of 579 (99.3%) |
| Max data gap | 700 ms (21 consecutive frames lost) | None |
| Camera consistency | ~38% of target rate | 98.3–98.7% |
Both recordings used identical SDK init parameters:
init_params.camera_resolution = sl::RESOLUTION::HD1080;
init_params.camera_fps = 30;
init_params.enable_image_enhancement = false;
init_params.camera_disable_self_calib = true;
init_params.enable_image_validity_check = 0;
init_params.depth_mode = sl::DEPTH_MODE::NONE;
We also limit concurrent retrieveImage(MEM::CPU) to 2 cameras at a time via a semaphore.
What We Observe
-
Frame drops happen at the
grab()level — our NVENC encoders report 0 dropped frames with queue_peak=1, so encoding is not the bottleneck -
All 4 cameras are affected roughly equally (~62% drops each), suggesting a shared resource bottleneck (GPU)
-
The drops only occur during camera motion with auto-exposure — stationary recording with auto-exposure has minimal drops
-
The Orin’s GPU governor (
nvhost_podgov) stays at 306 MHz (minimum) even under full ZED SDK load — it does not ramp toward the max 1300 MHz. We suspect this amplifies the issue, though locked exposure works fine at the same GPU clock.
Our Hypothesis
When the scene changes due to camera motion, auto-exposure/gain recalculation triggers additional GPU-side work inside grab(). At 306 MHz with 4 concurrent cameras, this pushes per-camera processing past the 33ms frame budget, causing frames to be dropped inside the SDK.
Our Current Workaround
Before recording, we read the current auto-determined exposure/gain/white-balance values via getCameraSettings(), switch to manual mode (AEC_AGC=0, WHITEBALANCE_AUTO=0), and write back the values. This gives us 0% drops at full 30 Hz. We restore auto mode when recording stops. The downside is that exposure cannot adapt to lighting changes between rooms during a recording.
Questions
-
Does
grab()perform additional GPU work when exposure/gain values change between frames (e.g., re-running ISP stages, histogram computation)? -
Is there a way to control the AEC convergence speed or update rate? (e.g., limit how fast exposure can change per frame)
-
Are there SDK parameters we’re missing that could reduce the GPU cost of auto-exposure in multi-camera setups?
-
Would constraining
AUTO_EXPOSURE_TIME_RANGEandAUTO_ANALOG_GAIN_RANGEto narrow ranges reduce the GPU work during scene transitions? -
Is there a way to get the GPU governor to ramp for ZED SDK workloads?
jetson_clockscauses a reboot on our unit (power budget issue).
System Info
-
Platform: Jetson AGX Orin 64GB (12-core ARMv8, 2048 CUDA cores)
-
OS: Ubuntu 22.04.3 LTS, L4T 36.4.4 (JetPack 6.2.1)
-
ZED SDK: 5.2.0
-
CUDA: 12.6 / Driver 540.4.0
-
Cameras: 4x ZED-XG 120mm (serials: 46527151, 46371805, 42181252, 48199258) on I2C buses 13 and 14
-
GMSL: MAX96712 deserializer, MAX9295 serializers, all 4 cameras detected and operational
Attached: ZED_Diagnostic_Results.json from our system.
We’d love to keep auto-exposure enabled during recording since lighting changes between rooms, but currently we must lock exposure before recording to avoid the 62% drop rate.
Thanks for any insights!
ZED_Diagnostic_Results.json: (I can’t upload files, and there is a limit on post content, so some logs are removed)
{
"AI Models": {
"info": [
"MULTI CLASS DETECTION is not optimized",
"MULTI CLASS MEDIUM DETECTION is not optimized",
"MULTI CLASS ACCURATE DETECTION is not optimized",
"HUMAN BODY FAST DETECTION is not optimized",
"HUMAN BODY MEDIUM DETECTION is not optimized",
"HUMAN BODY ACCURATE DETECTION is not optimized",
"HUMAN BODY 38 FAST DETECTION is not optimized",
"HUMAN BODY 38 MEDIUM DETECTION is not optimized",
"HUMAN BODY 38 ACCURATE DETECTION is not optimized",
"PERSON HEAD DETECTION is not optimized",
"PERSON HEAD ACCURATE DETECTION is not optimized",
"REID ASSOCIATION is not optimized",
"NEURAL LIGHT DEPTH is not optimized",
"NEURAL DEPTH is not optimized",
"NEURAL PLUS DEPTH is not optimized"
]
},
"CUDA Operations": {
"cudaDeviceCount": 1,
"cudaDriverVersion": 12060,
"cudaDriverVersionString": "12.6",
"cudaGetDeviceCountResult": 0,
"cudaRuntimeLoaded": true,
"cudaRuntimeVersion": 12060,
"cudaRuntimeVersionString": "12.6",
"cudaTestVersion": "1.0",
"gpuTests": [
{
"clockRateMHz": 1300,
"computeCapability": "8.7",
"computeCapabilityMajor": 8,
"computeCapabilityMinor": 7,
"freeResult": 0,
"freeSuccess": true,
"gpuId": 0,
"gpuName": "Orin",
"mallocResult": 0,
"mallocSuccess": true,
"memcpyD2HResult": 0,
"memcpyD2HSuccess": true,
"memcpyH2DResult": 0,
"memcpyH2DSuccess": true,
"memoryBusWidthBits": 256,
"memoryClockRateMHz": 1300,
"memsetResult": 0,
"memsetSuccess": true,
"multiProcessorCount": 16,
"pciBusId": 0,
"pciDeviceId": 0,
"pciDomainId": 0,
"setDeviceResult": 0,
"testElements": 1048576,
"testPassed": true,
"testSizeBytes": 4194304,
"totalGlobalMemMB": 62840,
"validationPassed": true
}
],
"testSizeMB": 1,
"valid": [
"<b>GPU 0 (Orin):</b> CUDA operations working correctly"
]
},
"GMSL DRIVER": {
"GMSL driver": "",
"error": [
"<b>No GMSL driver detected or not a Stereolabs board.</b>"
]
},
"Graphics Card": {
"deviceCount": 1,
"deviceDriverVersion": 12060,
"devices": [
{
"arch": "Ampere",
"computeCapability": "8.7",
"cores": 2048,
"name": "Orin",
"totalMemoryBytes": "65892741120",
"totalMemoryMB": 62840.21484375
}
],
"glx_info": "",
"initResult": 0,
"valid": [
"<b>Graphics card: </b> Orin"
]
},
"Log Kernel Trace (for ZED X)": {
"dmesg": {
"DMESG [DTS] LOG ": [
...
"[ 27.480832] zedx 13-0028: tegracam sensor driver:video0_v2.0.6",
"[ 27.567795] zedx 13-0028: Enable gmsl for channel 0 and id 0",
"[ 27.851032] tegra-camrtc-capture-vi tegra-capture-vi: subdev zedx 13-0028 bound",
"[ 27.852595] zedx 13-0028: Detected ZED-X sensor",
"[ 27.852603] zedx 13-0028: zedx_probe: Serial Number : 46371805",
"[ 27.852608] zedx 13-0028: zedx_probe: CHANNEL: 0 ID: 0",
"[ 27.853118] zedx 14-0011: Driver Version : v1.3.2",
"[ 27.853124] zedx 14-0011: Probing v4l2 sensor.",
"[ 27.853410] zedx 14-0011: tegracam sensor driver:video18_v2.0.6",
"[ 27.900664] zedx 14-0011: ZED-X detect error",
"[ 27.900702] zedx 14-0011: zedx_probe: ar0234 initialization failed",
"[ 27.900751] zedx: probe of 14-0011 failed with error -1",
"[ 27.900806] zedx 14-0010: Driver Version : v1.3.2",
"[ 27.900809] zedx 14-0010: Probing v4l2 sensor.",
"[ 27.901033] zedx 14-0010: tegracam sensor driver:video17_v2.0.6",
"[ 27.952965] zedx 14-0010: ZED-X detect error",
"[ 27.953004] zedx 14-0010: zedx_probe: ar0234 initialization failed",
"[ 27.953054] zedx: probe of 14-0010 failed with error -1",
"[ 27.953103] zedx 14-0030: Driver Version : v1.3.2",
"[ 27.953105] zedx 14-0030: Probing v4l2 sensor.",
"[ 27.953340] zedx 14-0030: tegracam sensor driver:video16_v2.0.6",
"[ 28.039574] zedx 14-0030: Enable gmsl for channel 0 and id 3",
"[ 28.317836] tegra-camrtc-capture-vi tegra-capture-vi: subdev zedx 14-0030 bound",
"[ 28.319253] zedx 14-0030: Detected ZED-X sensor",
"[ 28.319262] zedx 14-0030: zedx_probe: Serial Number : 42181252",
"[ 28.319270] zedx 14-0030: zedx_probe: CHANNEL: 0 ID: 3",
"[ 28.319974] zedx 14-0038: Driver Version : v1.3.2",
"[ 28.319978] zedx 14-0038: Probing v4l2 sensor.",
"[ 28.320113] zedx 14-0038: tegracam sensor driver:video15_v2.0.6",
"[ 28.407711] zedx 14-0038: Enable gmsl for channel 0 and id 3",
"[ 28.690057] tegra-camrtc-capture-vi tegra-capture-vi: subdev zedx 14-0038 bound",
"[ 28.691791] zedx 14-0038: Detected ZED-X sensor",
"[ 28.691798] zedx 14-0038: zedx_probe: Serial Number : 42181252",
"[ 28.691802] zedx 14-0038: zedx_probe: CHANNEL: 0 ID: 3",
"[ 28.692128] zedx 14-0020: Driver Version : v1.3.2",
"[ 28.692132] zedx 14-0020: Probing v4l2 sensor.",
"[ 28.692376] zedx 14-0020: tegracam sensor driver:video14_v2.0.6",
"[ 28.779501] zedx 14-0020: Enable gmsl for channel 0 and id 1",
"[ 29.053294] tegra-camrtc-capture-vi tegra-capture-vi: subdev zedx 14-0020 bound",
"[ 29.054008] zedx 14-0020: Detected ZED-X sensor",
"[ 29.054012] zedx 14-0020: zedx_probe: Serial Number : 48199258",
"[ 29.054014] zedx 14-0020: zedx_probe: CHANNEL: 0 ID: 1",
"[ 29.054322] zedx 14-0028: Driver Version : v1.3.2",
"[ 29.054324] zedx 14-0028: Probing v4l2 sensor.",
"[ 29.054447] zedx 14-0028: tegracam sensor driver:video13_v2.0.6",
"[ 29.139621] zedx 14-0028: Enable gmsl for channel 0 and id 1",
"[ 29.417820] tegra-camrtc-capture-vi tegra-capture-vi: subdev zedx 14-0028 bound",
"[ 29.419103] zedx 14-0028: Detected ZED-X sensor",
"[ 29.419107] zedx 14-0028: zedx_probe: Serial Number : 48199258",
"[ 29.419108] zedx 14-0028: zedx_probe: CHANNEL: 0 ID: 1",
"[ 29.428545] zedxone_uhd 9-001a: tegracam sensor driver:video32_v2.0.6",
"[ 29.429163] zedxone_uhd 9-001a: imx678_probe: imx678 initialization failed",
"[ 29.429374] zedxone_uhd 10-001a: tegracam sensor driver:video45_v2.0.6",
"[ 29.429747] zedxone_uhd 10-001a: imx678_probe: imx678 initialization failed",
"[ 29.430127] zedxone_uhd 13-001a: tegracam sensor driver:video6_v2.0.6",
"[ 29.431182] zedxone_uhd 13-001a: imx678_probe: imx678 initialization failed",
"[ 29.431746] zedxone_uhd 14-001a: tegracam sensor driver:video19_v2.0.6",
"[ 29.433099] zedxone_uhd 14-001a: imx678_probe: imx678 initialization failed"
],
"DTB list SLabs": [
"ls: cannot access '/boot/stereolabs/*': No such file or directory"
],
"DTB list original": [
"ls: cannot access '/boot/dtb/*': No such file or directory"
],
"Dpkg list Stereolabs": [
],
"Dpkg list zed": [
],
"Driver list": {
"/usr/lib/modules/5.15.148-rt-tegra/kernel/drivers/stereolabs/max9295/sl_max9295.ko": "not found",
"/usr/lib/modules/5.15.148-rt-tegra/kernel/drivers/stereolabs/max9296/max9296.ko": "not found",
"/usr/lib/modules/5.15.148-rt-tegra/kernel/drivers/stereolabs/max9296/sl_max9296.ko": "not found",
"/usr/lib/modules/5.15.148-rt-tegra/kernel/drivers/stereolabs/max96712/max96712.ko": "not found",
"/usr/lib/modules/5.15.148-rt-tegra/kernel/drivers/stereolabs/max96712/sl_max96712.ko": "not found",
"/usr/lib/modules/5.15.148-rt-tegra/kernel/drivers/stereolabs/zedone4k/sl_zedxone_uhd.ko": "not found",
"/usr/lib/modules/5.15.148-rt-tegra/kernel/drivers/stereolabs/zedx/max96712.ko": "not found",
"/usr/lib/modules/5.15.148-rt-tegra/kernel/drivers/stereolabs/zedx/sl_zedx.ko": "not found",
"/usr/lib/modules/5.15.148-rt-tegra/kernel/drivers/stereolabs/zedxpro/sl_zedxpro.ko": "not found"
},
"EXTlinux file": [
"cat: /boot/extlinux/extlinux.conf: No such file or directory"
],
"I2C Trace list": {
"I2C Trace /dev/i2C-0": [
" 0 1 2 3 4 5 6 7 8 9 a b c d e f",
"00: -- -- -- -- -- -- -- -- ",
"10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- ",
"20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- ",
"30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- ",
"40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- ",
"50: UU -- -- -- -- -- UU -- -- -- -- -- -- -- -- -- ",
"60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- ",
"70: -- -- -- -- -- -- -- -- "
],
....
},
"Jetpack": [
"# R36 (release), REVISION: 4.4, GCID: 41062509, BOARD: generic, EABI: aarch64, DATE: Mon Jun 16 16:07:13 UTC 2025",
"# KERNEL_VARIANT: oot",
"TARGET_USERSPACE_LIB_DIR=nvidia",
"TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia"
],
"Linux_uname": [
"Linux ubuntu 5.15.148-rt-tegra #1 SMP PREEMPT_RT Mon Oct 14 17:05:40 PDT 2024 aarch64 aarch64 aarch64 GNU/Linux"
],
"ZED SDK Version": {
"Diag": "5.2.0",
"RT": "5.2.0"
},
"ZEDX Daemon Status LS": [
"System has not been booted with systemd as init system (PID 1). Can't operate.",
"Failed to connect to bus: Host is down"
],
"ZEDX Driver State": [
"sl_zedxone_uhd 36864 0",
"sl_zedx 28672 0",
"sl_zedxpro 20480 0",
"sl_max9295 16384 3 sl_zedxpro,sl_zedxone_uhd,sl_zedx",
"sl_max96712 24576 4 sl_zedxpro,sl_zedxone_uhd,sl_max9295,sl_zedx",
"tegra_camera 217088 9 sl_zedxpro,nvhost_isp5,nvhost_nvcsi_t194,sl_zedxone_uhd,nv_hawk_owl,nvhost_vi5,sl_zedx",
"videodev 249856 7 v4l2_async,sl_zedxpro,videobuf2_v4l2,tegra_camera,sl_zedxone_uhd,videobuf2_common,sl_zedx"
],
"i2C Trace": [
"i2c-0\ti2c \t3160000.i2c \tI2C adapter",
"i2c-1\ti2c \tc240000.i2c \tI2C adapter",
"i2c-2\ti2c \t3180000.i2c \tI2C adapter",
"i2c-3\ti2c \t3190000.i2c \tI2C adapter",
"i2c-4\ti2c \tTegra BPMP I2C adapter \tI2C adapter",
"i2c-5\ti2c \t31b0000.i2c \tI2C adapter",
"i2c-6\ti2c \t31c0000.i2c \tI2C adapter",
"i2c-7\ti2c \tc250000.i2c \tI2C adapter",
"i2c-8\ti2c \t31e0000.i2c \tI2C adapter",
"i2c-9\ti2c \ti2c-2-mux (chan_id 0) \tI2C adapter",
"i2c-10\ti2c \ti2c-2-mux (chan_id 1) \tI2C adapter",
"i2c-11\ti2c \ti2c-2-mux (chan_id 2) \tI2C adapter",
"i2c-12\ti2c \ti2c-2-mux (chan_id 3) \tI2C adapter",
"i2c-13\ti2c \ti2c-8-mux (chan_id 0) \tI2C adapter",
"i2c-14\ti2c \ti2c-8-mux (chan_id 1) \tI2C adapter",
"i2c-15\ti2c \ti2c-8-mux (chan_id 2) \tI2C adapter",
"i2c-16\ti2c \ti2c-8-mux (chan_id 3) \tI2C adapter",
"i2c-17\ti2c \tNVIDIA SOC i2c adapter 0 \tI2C adapter"
]
}
},
"Processor": {
"L4T version": "36.4.4",
"OS": "Ubuntu 22.04.3 LTS ",
"avxSupported": false,
"coreCount": "12",
"cpu": " ARMv8 Processor rev 1 (v8l)",
"datetime": "2026-02-21 09:05:22",
"hyperThreading": "0",
"motherboard": "Jetson, NVIDIA",
"ramCapacity": "65.892741",
"ramUsage": "",
"ramUsed": "48.626012",
"threadCount": "12",
"valid": [
"<b>Processor: </b> ARMv8 Processor rev 1 (v8l)",
"<b>Motherboard: </b> Jetson, NVIDIA"
]
},
"USB Camera": {
"CorruptedFirmware": [
],
"GMSL driver": [
],
"GMSLList": [
{
"Serial": "46527151",
"Sub Model": "ZED-XG 120mm",
"id": 0,
"path": "/dev/i2c-13",
"state": "Available"
},
{
"Serial": "46371805",
"Sub Model": "ZED-XG 120mm",
"id": 1,
"path": "/dev/i2c-13",
"state": "Available"
},
{
"Serial": "46527151",
"Sub Model": "ZED-XG 120mm",
"id": 0,
"path": "/dev/i2c-13",
"state": "Available"
},
{
"Serial": "46371805",
"Sub Model": "ZED-XG 120mm",
"id": 1,
"path": "/dev/i2c-13",
"state": "Available"
},
{
"Serial": "42181252",
"Sub Model": "ZED-XG 120mm",
"id": 2,
"path": "/dev/i2c-14",
"state": "Available"
},
{
"Serial": "48199258",
"Sub Model": "ZED-XG 120mm",
"id": 3,
"path": "/dev/i2c-14",
"state": "Available"
},
{
"Serial": "42181252",
"Sub Model": "ZED-XG 120mm",
"id": 2,
"path": "/dev/i2c-14",
"state": "Available"
},
{
"Serial": "48199258",
"Sub Model": "ZED-XG 120mm",
"id": 3,
"path": "/dev/i2c-14",
"state": "Available"
}
],
"InternalDevicesCount (GMSL)": [
],
"InternalDevicesCount (USB)": [
],
"USBList": [
{
"USB_path": "/3",
"idProduct": "0x0420",
"idVendor": "0x0bda"
},
{
"USB_path": "/4",
"idProduct": "0x5420",
"idVendor": "0x0bda"
},
{
"USB_path": "/3",
"idProduct": "0x3549",
"idVendor": "0x13d3"
}
],
"ZEDCount": [
],
"ZEDs": [
],
"warning": [
"<b>No GMSL driver found.</b><br/> USB camera are not impacted by this.",
"<b>No GMSL driver found.</b><br/> USB camera are not impacted by this."
]
},
"ZED SDK": {
"CUDA Toolkit version": "V12.6.68 -->located in /usr/local/cuda-12.6/bin/nvcc",
"CUDA loading": false,
"GPU Driver Version": "Driver Version : 540.4.0",
"GPU Performance": " Performance State : N/A",
"ZED SDK Version (Diag)": "5.2.0",
"ZED SDK Version (RT)": "5.2.0",
"ai": {
"checkAI": true,
"cudaVersion": 12060,
"tensorVersion": 100300
},
"binFiles": [
"libsl_ai.so",
"libsl_zed.so"
],
"resourcesFiles": [
"neural_depth_5.3.model",
"neural_depth_light_5.2.model"
],
"resourcesFilesExpected": [
"objects_performance_3.2"
],
"valid": [
"<b>ZED SDK version:</b> 5.2.0",
"<b>CUDA version:</b> V12.6.68"
]
}
}