# ZED SDK 5.4.1 — dual ZED X Mini on GMSL: IMU sample delivery to one camera collapses to ~0.6 Hz while the Stereolabs kernel driver delivers ~206 Hz for both

Engineering escalation to Stereolabs. All figures below are measured on the platform
described in §2. Measurements were taken 2026-09-16.

---

## 1. Executive summary

With two ZED X Mini cameras streaming simultaneously on GMSL ports 0 and 1, the
application receives IMU samples for only one of the two cameras at the expected rate.
The other camera's IMU stream collapses to a fraction of a hertz, while the Stereolabs
BMI088 kernel driver is simultaneously acquiring that same camera's IMU at full rate
with zero I²C errors and a fully drained ring buffer.

**The core result, measured in a single 45 s dual-camera run:**

| layer | port 0 / SDK camera 0 | port 1 / SDK camera 1 |
|---|---|---|
| Stereolabs kernel driver `sample_count` (`/sys/class/bmi_spsc/`) | **~206 Hz** | **~209 Hz** |
| driver `i2c_errors` | 0 | 0 |
| driver `ring_status` | `head == tail`, `count = 0` (fully drained) | `head == tail`, `count = 0` (fully drained) |
| application, distinct IMU sample timestamps via ZED SDK | **~0.6 Hz** | **~200.6 Hz** |

Kernel acquisition: **~206 + ~209 Hz ≈ 415 Hz aggregate.**
SDK/application delivery: **~0.6 + ~200.6 Hz ≈ 201 Hz aggregate.**
Same run, same 45 seconds.

The kernel modules concerned (`bmi_spsc`, `sl_bmi088`) are **Stereolabs** modules, not
carrier-vendor modules, and their measured acquisition is healthy. The rings are being
fully drained, so a userspace consumer *is* reading those samples — but the application
never sees them for one of the two cameras.

Each camera individually is healthy: run alone, either camera delivers ~200 Hz to the
application. The failure requires concurrency.

The evidence localizes the loss to the Stereolabs userspace / ZED SDK 5.4.1 path,
between the working kernel acquisition layer and the public API surface. We cannot
identify the internal mechanism and do not claim to.

A secondary, independent defect is reported in §14: `SensorsData::imu.effective_rate`
returns physically impossible values (up to ~5.9e14 Hz) on a starved stream.

*Brief context only, not the subject of this report:* a gyroscope saturation problem we
observed on an earlier BSP/driver combination does **not** reproduce in this
environment. Stationary gyro noise is now ~0.1–0.17 °/s per-axis stddev against a
±1000 °/s full scale, with zero rail hits. That issue is considered resolved here.

---

## 2. Environment

| Item | Value |
|---|---|
| Compute module | NVIDIA Jetson Orin NX |
| Carrier | Connect Tech Hadron-GMSL (NGX018) |
| BSP | Connect Tech early-access, L4T R36.5.0 / JetPack 6.2.2 generation, CTI V004 |
| Kernel | 5.15.185-tegra |
| CUDA | 12.6 |
| ZED X driver | 1.4.3 |
| ZED SDK | 5.4.1 (`/usr/local/zed`) |
| Cameras | 2 × ZED X Mini (`MODEL::ZED_XM`), input type `GMSL` |
| Camera firmware | 2001 |
| Sensors firmware | 1000 |
| Power / governor | nvpmodel 40 W, 8 cores, `schedutil` |
| IMU kernel modules | `/lib/modules/5.15.185-tegra/updates/drivers/stereolabs/bmi088/{bmi_spsc,sl_bmi088}.ko` — Stereolabs-supplied, char major 486 |

IMU sensor configuration as reported by the SDK: accelerometer ±78.48 m/s² (±8 g),
gyroscope ±1000 °/s, both declared at **200 Hz**. No magnetometer, no barometer.

Both IMUs sit on I²C bus 2 at distinct addresses, with a deliberate 1250 µs sampling
phase offset, and are pinned to different CPUs (`target_cpu` 0 and 1). That design is
behaving as designed (§12).

Note on privileges: `/dev/spsc_bmi*` are root-only on this system, so the diagnostic is
run with sufficient privileges to open them. This is a permissions matter only — it is
not related to the delivery failure, and the failure is identical whenever the devices
open successfully.

---

## 3. Topology and serial mapping (fixed for all measurements below)

| serial | GMSL port | video node | SDK device id | IMU char device | I²C addrs (bus 2) |
|---|---|---|---|---|---|
| **58855396** | 0 | `/dev/video0` | 0 | `/dev/spsc_bmi0` | acc 0x19 / gyro 0x69 |
| **56680346** | 1 | `/dev/video2` | 1 | `/dev/spsc_bmi1` | acc 0x18 / gyro 0x68 |

This mapping is the **post-swap** configuration and was held constant for every result
in this document.

Prior to the swap, serial 56680346 occupied GMSL port 0 and **that** camera was the
starved one. After physically exchanging the two cameras between ports, serial
58855396 occupies port 0 and is now the starved one. The starvation stayed with the
port/SDK-index, not with the physical unit (§9).

---

## 4. Expected behavior

With two ZED X Mini cameras streaming concurrently, each `sl::Camera` instance should
deliver IMU samples to its own consumer at the sensor's declared 200 Hz, i.e. ~200 Hz
per camera, ~400 Hz aggregate — matching what the kernel driver demonstrably acquires.

## 5. Actual behavior

Dual operation: one camera delivers ~200 Hz of distinct IMU samples; the other delivers
0.3–2.3 Hz across the configurations tested. The **combined** distinct-sample rate is
pinned at ~201 Hz regardless of configuration (§11). Video is entirely unaffected in the
same runs: 0 grab failures, 0 LEFT/RIGHT retrieval failures, nominal FPS on both
cameras.

---

## 6. Minimal reproduction

1. Attach two ZED X Mini cameras to GMSL ports 0 and 1.
2. Open both as two independent `sl::Camera` instances, `DEPTH_MODE::NONE`, any
   resolution/FPS. (This mirrors the one-`sl::Camera`-per-thread pattern of the
   Stereolabs `samples/sensors_api/threaded` sample.)
3. In each camera's own thread, loop: `grab()`, then drain `getSensorsDataBatch()`.
4. Count **distinct** `imu.timestamp.data_ns` values per camera over 60 s.
   Discard repeats: the SDK returns the same stale `SensorsData` repeatedly, so any
   metric that does not deduplicate by timestamp will not show the fault.
5. **Do not use `imu.effective_rate` as the measurement** — see §14.

Concurrently, on the same host, read the driver counters (§16) to obtain the kernel-side
rate for the same interval.

Measurement methodology note: "distinct-sample rate" throughout this document means
unique IMU timestamps per second observed by the application. Every dual run also
recorded 0 grab failures and 0 image-retrieval failures.

---

## 7. Strongest evidence: same-run kernel vs. SDK

Reproduced from §1 because it is the central claim of this report. In one 45 s dual run:

- `spsc_bmi0` `sample_count` → **~206 Hz**; `spsc_bmi1` `sample_count` → **~209 Hz**
- `i2c_errors` = **0** on both; `ring_status` = `head == tail, count = 0` on both
- application distinct IMU samples: **~0.6 Hz** on camera 0, **~200.6 Hz** on camera 1

The Stereolabs kernel driver acquired ~415 Hz aggregate. The application, through the
ZED SDK public API, received ~201 Hz aggregate — and ~340× fewer distinct samples for
camera 0 than the driver produced for `spsc_bmi0` in the same interval.

Because `ring_status` shows both 32768-entry rings fully drained, the samples are being
consumed somewhere in userspace. They are not reaching the application.

---

## 8. Solo vs. dual

| run | port 0 distinct Hz | port 1 distinct Hz |
|---|---|---|
| camera 0 only, 1920×1200@60 | **199.9** | — |
| camera 1 only, 1920×1200@60 | — | **200.7** |
| both cameras, 1920×1200@60 | **2.0** | 199.2 |

Both cameras are healthy at ~200 Hz when operated individually. The trigger is
concurrent operation, not either unit.

Kernel-side, solo operation is equally healthy and the driver timer for the inactive
device is stopped:

| state | `spsc_bmi0` | `spsc_bmi1` |
|---|---|---|
| idle (both cameras closed) | 0 Hz, timer stopped | 0 Hz, timer stopped |
| camera 0 solo | 205 Hz, timer running | 0 Hz, timer stopped |
| camera 1 solo | 0 Hz, timer stopped | 208 Hz, timer running |
| both cameras | 206 Hz, timer running | 209 Hz, timer running |

---

## 9. Variables eliminated: physical camera and open order

**Physical camera — eliminated.** The two cameras were physically exchanged between
GMSL ports. Before the swap, serial 56680346 was on port 0 and starved. After the swap,
serial 58855396 is on port 0 and starves; 56680346, now on port 1, is healthy at
~200 Hz. The fault did not follow the physical unit.

**Open/start order — eliminated.** A `--reverse-open` mode opens and starts camera 1
first and camera 0 second, with identical settings, identical binary and identical
thread structure. Result: port 0 = 2.1 Hz, port 1 = 199.1 Hz, combined 200.8 Hz —
unchanged. The fault did not follow the open order.

**Still confounded — GMSL port 0 vs. SDK device index 0.** GMSL port 0, `/dev/video0`,
SDK device id 0 and `/dev/spsc_bmi0` are mutually confounded in this setup. The
measurements are equally consistent with "GMSL port 0 loses" and with "the device the
SDK enumerates first loses". Separating these would require a third camera or a way to
enumerate only the port-1 device as SDK index 0. **We do not claim the port has been
independently isolated from the SDK index.**

---

## 10. Independence from video workload

Dual operation, 60 s per run, LEFT+RIGHT retrieved, no recording:

| mode | port 0 Hz | port 1 Hz | distinct samples (p0 + p1) | combined Hz |
|---|---|---|---|---|
| 1920×1200 @ 60 | 2.0 | 199.2 | 99 + 11956 | 200.9 |
| 1920×1200 @ 30 | 2.3 | 198.8 | 117 + 11935 | 200.9 |
| 1920×1200 @ 15 | 0.6 | 200.2 | 33 + 12019 | 200.9 |
| 1920×1080 @ 30 | 2.2 | 198.9 | 112 + 11942 | 200.9 |
| 960×600 @ 120 | 2.0 | 199.5 | 106 + 11952 | 201.0 |
| 960×600 @ 60 | 0.3 | 200.6 | 20 + 12040 | 201.0 |
| 960×600 @ 30 | 1.9 | 199.2 | 100 + 11956 | 200.9 |
| 960×600 @ 15 | 0.3 | 200.5 | 20 + 12038 | 201.0 |

Pixel throughput varies **16×** across this matrix and grab rate varies **8×**. Port-0
delivery does not improve; the *lowest*-workload modes are among the worst. Every run:
0 grab failures, 0 retrieval failures, video FPS at nominal.

Retrieval and recording, at 1920×1200@60 dual:

| variant | port 0 Hz | port 1 Hz | combined Hz |
|---|---|---|---|
| grab + retrieve LEFT/RIGHT + IMU | 2.0 | 199.2 | 200.9 |
| grab + retrieve + SVO2 H.265 recording + IMU | 2.2 | 199.1 | 200.9 |
| grab + IMU only (no image retrieval) | **31.1** | **170.2** | 200.9 |

SVO2 recording changes nothing. Removing image retrieval **redistributes** the budget —
port 0 gains 1756 distinct samples, port 1 loses 1757 — without changing the total.

---

## 11. The ~201 Hz combined userspace ceiling

Across all twelve dual configurations measured, total distinct samples per 60 s fall in
the range **12052–12060** (200.9–201.0 Hz), with a standard deviation under 0.05 Hz.
Solo, a single camera receives the full ~200 Hz.

The ceiling is approximately the rate of **one** IMU, not two, and it is zero-sum: the
samples one camera loses appear on the other (§10, grab-only row). The kernel is
producing ~415 Hz aggregate in the same runs. The ceiling therefore appears at or below
the SDK API boundary, not at the sensor, bus or driver.

---

## 12. Kernel driver and hardware evidence

`bmi_spsc` (char major 486) exposes per-device counters under
`/sys/class/bmi_spsc/spsc_bmi{0,1}/`. Measured over 10 s windows (rates in §8):

- **`sample_count`** — increments at 205–209 Hz for every active device, in solo and in
  dual operation alike. No degradation under concurrency.
- **`i2c_errors`** — **0** on both devices in every state measured.
- **`ring_status`** — `head == tail, count = 0` on both devices: the 32768-entry rings
  are fully drained by their consumer. There is no backlog and no overflow.
- **I²C controller interrupts** (`3180000.i2c`, IRQ 115) scale with the number of active
  IMUs and not with video workload: 0/s idle → ~1067/s and ~1239/s with each camera solo
  → ~1968–2179/s with both cameras active. Consistent with both IMUs being polled at
  full rate concurrently.
- **`dmesg`** shows no spsc, bmi, GMSL or I²C errors at any point during any run.
- The `zed_x_daemon` is not in the IMU path in this configuration: it is idle (0.1% CPU)
  and the IMU character devices are opened by the application process itself.

Every layer from the sensor through I²C, the Stereolabs kernel driver and its ring
buffer behaves correctly and at full rate for both devices simultaneously.

---

## 13. SDK API behavior

### 13.1 `getSensorsDataBatch()` returns SUCCESS with an empty vector

Instrumented accounting over a 1920×1200@60 dual run:

| | port 0 | port 1 |
|---|---|---|
| batch calls | 3622 | 7158 |
| calls returning ≥1 sample | 42 | 3579 |
| calls returning **SUCCESS with an empty vector** | **3580** | 3579 |
| calls returning an error | **0** | 0 |
| total samples obtained from batch | 99 | 11956 |
| fell back to `getSensorsData()` | 3538 | 0 |

Port 0's batch call is empty on **98.8%** of invocations and **never** returns an error
code. From the application's point of view the API reports complete success while
delivering nothing.

When port 0 does deliver, it delivers a burst of exactly **20 samples** spaced ~6.2 ms
apart (≈161 Hz *within* the burst), roughly every 10 s. The longest observed gap between
distinct port-0 samples was **21.5 s**. The fixed burst size and the internal spacing
look more like a periodic flush of a stale queue than like rate throttling, but we
cannot determine the mechanism.

Port 1's empty-batch count is similar in absolute terms, but that is a property of the
drain loop, which calls until the batch comes back empty: port 1 alternates non-empty and
empty calls and its non-empty calls carry the full sample stream. Port 0's 42 non-empty
calls carry 99 samples in total.

### 13.2 `sl::Sensors` reproduces the failure identically

`sl::Sensors` (the SDK 5.4.1 multi-device sensor manager) is a distinct code path: no
`sl::Camera` batch retrieval, `getMotionSensorsData()` polled from a dedicated thread
per device at ~1.7 kHz (≈8.5× the sensor rate, so undersampling by the measurement is
excluded).

| `sl::Sensors` run | port 0 Hz | port 1 Hz | poll errors |
|---|---|---|---|
| 1920×1200 @ 60, retrieval on | **2.00** (120 distinct) | 189.9 | 0 |
| 960×600 @ 15, retrieval off | **2.18** (131 distinct) | 193.1 | 0 |

The 120 distinct samples at 1920×1200@60 is **identical** to the `sl::Camera` control at
the same setting. Every poll returns `SUCCESS`; between distinct updates the API returns
the same stale sample repeatedly.

**`sl::Sensors` is therefore not a workaround.** Two unrelated public APIs fail in the
same way, at the same magnitude, on the same device.

---

## 14. Separate SDK issue: `imu.effective_rate` returns invalid values

On a starved stream actually delivering 0.9–2.9 Hz of distinct samples,
`SensorsData::imu.effective_rate` returned values including:

- `2294986833920.0` Hz
- `588480849641472.0` Hz

On healthy streams the same field reports a plausible 200–204 Hz.

These values are not physically meaningful and are not a scaled version of the true
rate. This is independently reportable and is the reason `effective_rate` cannot be used
to measure this fault — any tool relying on it will not detect the starvation.

---

## 15. Ruled out, and what remains open

### Ruled out by measurement

| Hypothesis | Basis |
|---|---|
| Physical camera / individual unit defect | Port swap; fault stayed with the port/index. Both units healthy solo at ~200 Hz (§8, §9) |
| Camera open / start order | `--reverse-open`: unchanged (§9) |
| Video resolution | 16× pixel-throughput range, no effect (§10) |
| Video frame rate | 8× grab-rate range, no effect; 15 FPS among the worst (§10) |
| Image retrieval | Removing it redistributes but does not increase the total (§10) |
| SVO2 / H.265 recording | No effect (§10) |
| CPU saturation | §15.1 |
| GPU / NVENC saturation | §15.1 |
| Memory-bandwidth (EMC) saturation | §15.1 |
| Kernel driver acquisition rate | 205–209 Hz per device, solo and dual (§12) |
| I²C bus errors / bus capacity | `i2c_errors` = 0; IRQ rate scales cleanly with active IMU count (§12) |
| Ring buffer overflow / backlog | `ring_status`: `head == tail, count = 0` (§12) |
| Kernel/driver error reporting | `dmesg` silent throughout (§12) |
| Application architecture or API misuse | Matches the Stereolabs threaded sensors sample; a second, unrelated API reproduces it (§13.2) |
| `zed_x_daemon` involvement | Idle at 0.1% CPU; not in the IMU path here (§12) |
| Non-monotonic IMU timestamps as a cause | Not correlated: the highest backwards-timestamp count observed (54 per 60 s) was on a camera running **solo at a healthy 199.9 Hz**; the starved `sl::Sensors` device recorded **0**. A separate, minor issue |

### 15.1 Resource utilization — nothing is saturated

| config | total CPU | busiest core | app process | ZED daemon | GPU (GR3D) | EMC | port 0 Hz |
|---|---|---|---|---|---|---|---|
| 1920×1200@60 dual, retrieval on | 22.5% mean / 24.6% max | 33.0% max | 80% of one core (of 800% available) | 0.1% | 60.9% mean / 79% max | 32.5% | 2.0 |
| 960×600@15 dual, grab only | 6.2% mean / 20.5% max | — | 11% of one core | 0.1% | 6.7% mean | 1.0% | 0.9 |

No core is saturated in either case. Reducing CPU ~10×, GPU ~9× and memory bandwidth
~32× makes port-0 delivery slightly **worse**, not better. This is inconsistent with any
contention or scheduling-pressure explanation at the system-resource level.

### Open

- **O1.** GMSL port 0 vs. SDK device index 0 remain confounded (§9). Not separable with
  two cameras.
- **O2.** The exact userspace mechanism is unknown. We cannot distinguish between a
  shared reader thread, a per-process lock, a mis-keyed device→consumer mapping, a
  shared dispatch/publication slot, or something else, without SDK sources or a
  Stereolabs debug build.
- **O3.** Why port 0 delivers in fixed 20-sample bursts every ~10 s (§13.1).
- **O4.** Whether more than two cameras degrade further — untested; only two cameras
  available.

---

## 16. Commands to reproduce the driver-side counters

During an active dual-camera run, on the Jetson:

```bash
# Per-device IMU acquisition counters exposed by the Stereolabs bmi_spsc module.
# Sample twice, N seconds apart, and divide the delta by N to get the kernel-side rate.
cat /sys/class/bmi_spsc/spsc_bmi0/sample_count
cat /sys/class/bmi_spsc/spsc_bmi1/sample_count

# Ring occupancy: head == tail, count = 0 means the ring is being fully drained.
cat /sys/class/bmi_spsc/spsc_bmi0/ring_status
cat /sys/class/bmi_spsc/spsc_bmi1/ring_status

# I2C error counters: 0 on both throughout.
cat /sys/class/bmi_spsc/spsc_bmi0/i2c_errors
cat /sys/class/bmi_spsc/spsc_bmi1/i2c_errors

# I2C controller interrupt rate (sample twice, N seconds apart).
grep '3180000.i2c' /proc/interrupts

# Kernel log, checked for spsc / bmi / GMSL / I2C messages.
dmesg -T | tail -n 200
```

Application-side counting pattern (one such thread per camera):

```cpp
// Per camera: sl::Camera cam; InitParameters with depth_mode = DEPTH_MODE::NONE.
// Count DISTINCT imu.timestamp.data_ns. Repeats must be discarded, or the fault
// is invisible. Do not use imu.effective_rate (see section 14).

uint64_t last_ts = 0, distinct = 0, dup = 0;
uint64_t batch_calls = 0, batch_empty_success = 0, batch_errors = 0;

while (running) {
    if (cam.grab() != sl::ERROR_CODE::SUCCESS) continue;

    for (int drain = 0; drain < 8; ++drain) {
        std::vector<sl::SensorsData> batch;
        const sl::ERROR_CODE e = cam.getSensorsDataBatch(batch);
        ++batch_calls;
        if (e != sl::ERROR_CODE::SUCCESS) { ++batch_errors; break; }
        if (batch.empty())               { ++batch_empty_success; break; }

        for (const sl::SensorsData& sd : batch) {
            if (!sd.imu.is_available) continue;
            const uint64_t ts = sd.imu.timestamp.data_ns;
            if (ts == last_ts) { ++dup; continue; }
            ++distinct;
            last_ts = ts;
        }
    }
}
// distinct / elapsed_seconds is the authoritative per-camera IMU delivery rate.
```

The `sl::Sensors` reproduction (§13.2) uses `sensors.init()`, `sensors.add()` per camera
with `input.setFromCameraID(dev)`, and one thread per device polling
`sensors.getMotionSensorsData(sd, sl::TIME_REFERENCE::CURRENT, id)` at ~1.7 kHz, again
counting distinct `imu.timestamp.data_ns`.

---

## 17. Conclusion

Stated to the limit of the evidence and no further:

1. **Measured fact.** In the same dual-camera run, the Stereolabs `bmi_spsc` kernel
   driver acquires ~206 Hz and ~209 Hz on the two IMU devices, with 0 I²C errors and
   both ring buffers fully drained, while the application receives ~0.6 Hz and
   ~200.6 Hz of distinct IMU samples through the ZED SDK.

2. **Measured fact.** Both cameras deliver ~200 Hz individually. The combined dual-camera
   delivery is pinned at ~201 Hz across every configuration tested, and is zero-sum
   between the two cameras.

3. **Supported inference.** Because kernel acquisition is healthy and the rings are being
   drained, and because two unrelated public SDK APIs (`sl::Camera` +
   `getSensorsDataBatch()`, and `sl::Sensors` + `getMotionSensorsData()`) fail
   identically, the sample loss is located in the Stereolabs userspace / ZED SDK 5.4.1
   path — below the public API surface and above the kernel acquisition layer.

4. **Explicitly not claimed.** We do not know which internal SDK component is
   responsible, nor the mechanism. We have not separated GMSL port 0 from SDK device
   index 0 (§9, O1); those remain confounded and no claim is made about which of the two
   determines the outcome.

5. **Not implicated by the present evidence.** The kernel driver, the I²C bus, the GMSL
   video path, the carrier BSP's IMU integration, system resource limits, the ZED daemon,
   the application's threading architecture, and the application's API usage.

---

## 18. Requested from Stereolabs

1. Is IMU sample dispatch from `bmi_spsc` to `sl::Camera` / `sl::Sensors` consumers
   served by a single shared thread, slot or buffer with an aggregate budget on the order
   of 200 samples/s, rather than per-device paths? The observed ~201 Hz zero-sum ceiling
   would be consistent with that.
2. Is concurrent multi-camera IMU capture on GMSL validated at full rate for ≥2 ZED X /
   ZED X Mini devices on ZED SDK 5.4.1? If so, on what carrier and BSP?
3. Can Stereolabs reproduce two ZED X Mini devices simultaneously delivering ~200 Hz of
   distinct IMU samples each on any platform? A confirmed-working configuration would
   immediately narrow this.
4. Why does `getSensorsDataBatch()` return `SUCCESS` with an empty vector — never an
   error or a status — on 98.8% of calls for a starved stream? Is there a status or
   diagnostic surface that would let an application detect this condition?
5. What produces the fixed 20-sample burst every ~10 s on the starved stream (§13.1)?
6. Is `imu.effective_rate` expected to be valid when a stream is starved? The observed
   values (up to ~5.9e14 Hz) appear to be a distinct defect (§14).
7. Is there a newer SDK build, patch or driver revision addressing multi-camera IMU
   delivery that we can test?
8. Is there a supported workaround at full rate? `sl::Sensors` is not one (§13.2).
9. Would Stereolabs like the full internal investigation report, the diagnostic
   application source, and the raw per-run measurement data? All are available on
   request.