ZED X One: IMU temperature read livelocks CPU 0 for hours on the RT kernel (rpm_resume in tegra_i2c_xfer), ending in a hard reset

Hardware ZED X
,
1

Setup

  • Jetson AGX Orin, L4T R36.4.7, RT kernel 5.15.148-rt-tegra (CONFIG_PREEMPT_RT=y)
  • ZED Link Duo driver stereolabs-zedlink-duo 1.4.3-LI-MAX96712-rt-L4T36.4.0
  • One ZED X One 4K (zedxone_uhd), ZED SDK 5.4.1, zed-ros2-wrapper 5.4.1 (zed_camera_one component)

What happens

Every few hours to days, the zed_node_main process gets stuck in a kernel I2C read of the IMU temperature (/sys/class/bmi_spsc/spsc_bmi0/accel_temperature). It doesn’t recover. CPU 0 stays busy in that thread, and the kernel reports a new rcu_preempt self-detected stall on CPU every 63 s for 1 to 4 hours, until the Jetson hard-resets with no panic message.

On this module every interrupt is handled by CPU 0: the I350 PCIe Ethernet, Wi-Fi, NVMe, the USB host controller and the I2C controller. So the whole system degrades during the stall:

  • the Ethernet and Wi-Fi links stop answering;
  • USB devices disconnect;
  • the last minutes of the logs never reach the disk.

Since we installed ZED Link 1.4.3 on 2026-09-08, kern.log has collected about 5,700 such stall reports. Every one has this stack:

rcu: INFO: rcu_preempt self-detected stall on CPU
rcu:    0-....: (1 GPs behind) idle=30f/1/0x4000000000000002 softirq=0/0 fqs=2120
Task dump for CPU 0:
task:zed_node_main   state:R  running task     stack:    0 pid: 9707 ppid:  4620 flags:0x0000000c
 ...
 el1h_64_irq+0x7c/0x80
 rt_spin_unlock+0x44/0xb0
 rpm_resume+0x9c/0x730
 __pm_runtime_resume+0x44/0xb0
 tegra_i2c_xfer+0x54/0x880
 __i2c_transfer+0x1dc/0x860
 i2c_smbus_xfer_emulated+0x114/0x700
 __i2c_smbus_xfer+0x134/0x580
 pca954x_select_chan+0x88/0xe0
 __i2c_mux_master_xfer+0x4c/0xb0
 __i2c_transfer+0x1dc/0x860
 i2c_transfer+0xa0/0x130
 bmi_i2c_rd+0x90/0xf8 [bmi088]
 bmi_temp+0x70/0xf8 [bmi088]
 accel_temperature_show+0x68/0x100 [bmi_spsc]
 dev_attr_show+0x44/0xa0
 sysfs_kf_seq_show+0x94/0x160
 ...
 ksys_read+0x7c/0x110

The I2C bus is 3180000.i2c, behind the pca9546 mux, on the channel holding sl_max96712, zedxone_uhd and bmi088.

What we think is going on

These are our own conclusions; we haven’t confirmed them in the driver source.

  1. The reading thread is real-time. zed_node_main contains two SCHED_FIFO threads, at priorities 99 and 50, both pinned to CPU 0. The wrapper didn’t create them: its advanced.thread_sched_policy is the default SCHED_BATCH and it logs “Change thread priority: FALSE”. We assume libsl_zed.so creates them; it does call pthread_setschedparam and pthread_setaffinity_np.
  2. The wake-up becomes a busy-wait. tegra_i2c_xfer calls pm_runtime_resume on an irq-safe device. If the controller is in the middle of autosuspending, rpm_resume busy-waits (the rt_spin_unlock / cpu_relax loop seen in the trace) for the suspend to finish.
  3. On PREEMPT_RT the suspend can’t finish. The suspend runs in a preemptible context. With a priority-99 thread spinning on the same CPU, it never gets to run, so the read never returns: a livelock.
  4. It’s rare and random because the temperature read has to land exactly in an autosuspend window. The IMU daemon’s 200 Hz traffic on the same bus makes those windows frequent.

What we did

  • We stopped the wrapper’s 1 Hz temperature polling when publish_temp is off. That made the stalls rarer, but they still happen, because something inside the SDK still reads the temperature.
  • Workaround: keep the I2C controller out of runtime suspend:
    echo on | sudo tee /sys/bus/platform/devices/3180000.i2c/power/control
    It’s been applied and we’re watching for new stalls.

Questions

  1. Does the SDK read the IMU temperature, through accel_temperature, from its priority-99 thread pinned to CPU 0? Can that priority or CPU pinning be configured?
  2. Could the ZED Link driver avoid runtime autosuspend of this I2C controller on RT kernels, or read the temperature in a way that can’t spin?
  3. Is this known, and is a fix planned?

We can provide full logs if useful.

2

Hi @deepanaishtaweera,
Welcome to the StereoLabs community

Thank you for the detailed report and for the analysis of the stack trace; it’s very helpful.

Yes, the ZED SDK reads the IMU temperature together with the other sensor data, so disabling the wrapper’s temperature polling reduces the accesses but can’t remove them. The priority and CPU affinity of the SDK’s internal threads can’t be configured in ZED SDK v5.4.1.

This isn’t a known issue. Your analysis of the rpm_resume busy-wait under PREEMPT_RT is consistent with the trace, and I’ve reported it to the ZED Link driver team so they can evaluate keeping the 3180000.i2c controller out of runtime suspend in the RT driver builds. In the meantime, your workaround (echo on > /sys/bus/platform/devices/3180000.i2c/power/control) is the right approach. To make it persistent across reboots, you can apply it with a udev rule or a systemd service.

Please share the full kern.log of a stall episode and the output of sudo journalctl -u zed_x_daemon from the same boot. Please also let us know whether new stalls appear with the workaround applied.