ZEDmini on Nvidia Orin with USB Problem

Hello,
We are using the ZED Mini alongside a (discontinued) Microsoft Azure Kinect as a Stereo-Vision + ToF pair in ROS2 on a Nvidia Jetson Orin AGX.
We connect the Cameras, so that both have their own 10GBit/s USB master-port.

$ lsusb -tv
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 10000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
    |__ Port 1: Dev 6, If 0, Class=Video, Driver=uvcvideo, 5000M
        ID 2b03:f682 STEREOLABS ZED-M camera
    |__ Port 1: Dev 6, If 1, Class=Video, Driver=uvcvideo, 5000M
        ID 2b03:f682 STEREOLABS ZED-M camera
    |__ Port 3: Dev 2, If 0, Class=Hub, Driver=hub/4p, 10000M
        ID 0bda:0420 Realtek Semiconductor Corp.
        |__ Port 3: Dev 3, If 0, Class=Hub, Driver=hub/2p, 5000M
            ID 045e:097a Microsoft Corp.
            |__ Port 1: Dev 4, If 2, Class=Vendor Specific Class, Driver=, 5000M
                ID 045e:097d Microsoft Corp.
            |__ Port 1: Dev 4, If 0, Class=Video, Driver=, 5000M
                ID 045e:097d Microsoft Corp.
            |__ Port 1: Dev 4, If 1, Class=Video, Driver=, 5000M
                ID 045e:097d Microsoft Corp.
            |__ Port 2: Dev 5, If 0, Class=Vendor Specific Class, Driver=, 5000M
                ID 045e:097c Microsoft Corp.
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
    |__ Port 2: Dev 8, If 0, Class=Human Interface Device, Driver=usbfs, 12M
        ID 2b03:f681 STEREOLABS ZED-M HID Interface
    |__ Port 3: Dev 3, If 0, Class=Wireless, Driver=rtk_btusb, 12M
        ID 13d3:3549 IMC Networks
    |__ Port 3: Dev 3, If 1, Class=Wireless, Driver=rtk_btusb, 12M
        ID 13d3:3549 IMC Networks
    |__ Port 4: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M
        ID 0bda:5420 Realtek Semiconductor Corp.
        |__ Port 3: Dev 6, If 0, Class=Hub, Driver=hub/4p, 480M
            ID 045e:097b Microsoft Corp.
            |__ Port 3: Dev 7, If 1, Class=Audio, Driver=snd-usb-audio, 480M
                ID 045e:097e Microsoft Corp.
            |__ Port 3: Dev 7, If 2, Class=Human Interface Device, Driver=usbhid, 480M
                ID 045e:097e Microsoft Corp.
            |__ Port 3: Dev 7, If 0, Class=Audio, Driver=snd-usb-audio, 480M
                ID 045e:097e Microsoft Corp.
        |__ Port 2: Dev 5, If 2, Class=Human Interface Device, Driver=usbhid, 12M
            ID 046d:c52b Logitech, Inc. Unifying Receiver
        |__ Port 2: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 12M
            ID 046d:c52b Logitech, Inc. Unifying Receiver
        |__ Port 2: Dev 5, If 1, Class=Human Interface Device, Driver=usbhid, 12M
            ID 046d:c52b Logitech, Inc. Unifying Receiver

I set up the ZEDmini to run at FullHD with 30FPS in the ROS2-Config File and start it with the following command:

ros2 launch zed_wrapper zed_camera.launch.py camera_model:=zedm

This works without issues. RVIZ2 shows the fluid video stream of both cameras plus the depth map und so on. No problems until now.
But: When I start the Azure Kinect alongside with the following command:

ros2 launch azure_kinect_ros_driver driver.launch.py color_resolution:=1080P depth_mode:=NFOV_UNBINNED fps:=30

The ZEDmini loses its connection, while the Azure Kinect runs.

[zed_wrapper-2] [ERROR] [1724737690.468687269] [zed.zed_node]: Connection issue detected: CAMERA_REBOOTING
[zed_wrapper-2] [ERROR] [1724737691.708659450] [zed.zed_node]: Connection issue detected: CAMERA_REBOOTING
[zed_wrapper-2] [ERROR] [1724737694.454848273] [zed.zed_node]: Connection issue detected: CAMERA_REBOOTING

(this goes on indefinitely)

Then again killing the ZEDmini driver and restarting it, causes this behaviour:

[zed_wrapper-2] [2024-08-27 05:57:22 UTC][ZED][INFO] [Init]  Waiting 728 msec before next opening attempt.
[zed_wrapper-2] [2024-08-27 05:57:23 UTC][ZED][INFO] [Init]  Trying to force a device reboot to recover the video module.
[zed_wrapper-2] [2024-08-27 05:57:25 UTC][ZED][INFO] [Init]  Timeout in 129msec
[zed_wrapper-2] [2024-08-27 05:57:25 UTC][ZED][INFO] [Init]  Opening Camera. Attempt #3
[zed_wrapper-2] [2024-08-27 05:57:26 UTC][ZED][ERROR] [Init]  Unable to capture images. Consider trying a lower resolution and/or FPS
[zed_wrapper-2] [2024-08-27 05:57:26 UTC][ZED][INFO] [Init]  Waiting 1385 msec before next opening attempt.
[zed_wrapper-2] [2024-08-27 05:57:27 UTC][ZED][INFO] [Init]  Trying to force a device reboot to recover the video module.
[zed_wrapper-2] [2024-08-27 05:57:30 UTC][ZED][INFO] [Init]  Camera opening timeout reached.
[zed_wrapper-2] [2024-08-27 05:57:30 UTC][ZED][WARNING] LOW USB BANDWIDTH in sl::ERROR_CODE sl::Camera::open(sl::InitParameters)
[zed_wrapper-2] [WARN] [1724738250.355916612] [zed.zed_node]: Error opening camera: LOW USB BANDWIDTH
[zed_wrapper-2] [INFO] [1724738250.356054079] [zed.zed_node]: Please verify the camera connection

But: The Azure Kinect works just fine in this state - almost as it would hog some resources the ZED cannot access anymore.

The Azure Kinect works without issues, but it seems that the ZEDmini gets somehow starved of bandwidth.
Interestingly, when lowering both camera’s resolutions to 720p, they work as intended. Increasing either the Azure or the ZEDmini to 1080p, the ZEDmini (and only the ZEDmini) starts stuttering.

Test Case Status
Single ZEDmini OK
Single Azure Kinect OK
Both (FullHD) ZEDmini fails with “Connection Issue” / “Low USB bandwidth”
Both (ZEDmini@FullHD, AzureKinect@720p) Stuttering on ZEDmini (inconsistent Frame rate + weird vertical screen-tearing)
Both (ZEDmini@720P, AzureKinect@FullHD) Stuttering on ZEDmini - but less harsh
Both (720P) OK

Note: No camera gets unplugged - the hardware setup remains unchanged. There are no changes in the lsusb-command. It is only a matter of if the cameras are running and what resolution they are streaming.

Running ZEDDiagnostics has two states:
If the Azure Kinect is Running at FullHD:

Summary
{
    "AI Models": {
        "info": [
            "MULTI CLASS DETECTION is not optimized",
            "MULTI CLASS MEDIUM DETECTION is not optimized",
            "MULTI CLASS ACCURATE DETECTION is not optimized",
            "HUMAN BODY FAST DETECTION is not optimized",
            "HUMAN BODY MEDIUM DETECTION is not optimized",
            "HUMAN BODY ACCURATE DETECTION is not optimized",
            "HUMAN BODY 38 FAST DETECTION is not optimized",
            "HUMAN BODY 38 MEDIUM DETECTION is not optimized",
            "HUMAN BODY 38 ACCURATE DETECTION is not optimized",
            "PERSON HEAD DETECTION is not optimized",
            "PERSON HEAD ACCURATE DETECTION is not optimized",
            "REID ASSOCIATION is not optimized",
            "NEURAL DEPTH is not optimized",
            "NEURAL PLUS DEPTH is not optimized"
        ]
    },
    "Camera Test": {
        "InternalDevicesCount": 2,
        "ZEDCount": 1,
        "camera": {
            "Device ID": 0,
            "Firmware": 1523,
            "Init Output": "OK",
            "Model": "ZED Mini",
            "Serial Number": 10027879,
            "valid": [
                "<b>Camera: </b>ZED Mini",
                "<b>Firmware: </b>1523"
            ]
        },
        "error": [
            "<b>Specific resolutions are not available</b> <br/> Replug your device or connect the camera to another <b>USB 3.0</b> port."
        ],
        "resolutions": {
            "HD1080@15": {
                "fps (input)": 15,
                "fps (output)": 15,
                "height (input)": 1080,
                "height (output)": 1080,
                "initialization": "OK",
                "status": true,
                "width (input)": 1920,
                "width (output)": 1920
            },
            "HD1080@30": {
                "fps (input)": 30,
                "fps (output)": 26,
                "height (input)": 1080,
                "height (output)": 1080,
                "initialization": "OK",
                "status": true,
                "width (input)": 1920,
                "width (output)": 1920
            },
            "HD2K@15": {
                "initialization": "Unknown",
                "status": false
            },
            "HD720@15": {
                "fps (input)": 15,
                "fps (output)": 15,
                "height (input)": 720,
                "height (output)": 720,
                "initialization": "OK",
                "status": true,
                "width (input)": 1280,
                "width (output)": 1280
            },
            "HD720@30": {
                "fps (input)": 30,
                "fps (output)": 30,
                "height (input)": 720,
                "height (output)": 720,
                "initialization": "OK",
                "status": true,
                "width (input)": 1280,
                "width (output)": 1280
            },
            "HD720@60": {
                "dropCount": 248,
                "fps (input)": 60,
                "fps (output)": 31,
                "height (input)": 720,
                "height (output)": 720,
                "imageCount": 252,
                "initialization": "OK",
                "status": true,
                "tearingCount": 11,
                "width (input)": 1280,
                "width (output)": 1280
            },
            "VGA@100": {
                "fps (input)": 100,
                "fps (output)": 86,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            },
            "VGA@15": {
                "fps (input)": 15,
                "fps (output)": 15,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            },
            "VGA@30": {
                "fps (input)": 30,
                "fps (output)": 30,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            },
            "VGA@60": {
                "fps (input)": 60,
                "fps (output)": 60,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            }
        },
        "sensors": {
            "Firmware": 515,
            "Init Output": "OK",
            "Model": "ZED Mini (MCU)",
            "Serial Number": 10027879
        }
    },
    "Devices": {
        "CorruptedFirmware": false,
        "GMSL driver": "",
        "GMSL driver compatiblity": "Unknown",
        "GMSLList": [
        ],
        "MCUDetected": true,
        "USBList": [
            {
                "USB_path": "/3/3/2",
                "idProduct": "0x097c",
                "idVendor": "0x045e"
            },
            {
                "USB_path": "/3/3/1",
                "idProduct": "0x097d",
                "idVendor": "0x045e"
            },
            {
                "USB_path": "/3/3",
                "idProduct": "0x097a",
                "idVendor": "0x045e"
            },
            {
                "USB_path": "/3",
                "idProduct": "0x0420",
                "idVendor": "0x0bda"
            },
            {
                "USBMode": 3,
                "USB_path": "/1",
                "bDescriptorType": 1,
                "bDeviceProtocol": 1,
                "bLength": 18,
                "bMaxPacketSize0": 9,
                "bNumConfigurations": 1,
                "bcdDevice": "1.0",
                "bcdUSB": "3.0",
                "bcdUSBClass": 239,
                "bcdUSBSubClass": 2,
                "busNumber": 2,
                "device": "ZED mini",
                "iManufacturer": 1,
                "iProduct": 2,
                "iSerial": 0,
                "idProduct": "0xf682",
                "idVendor": "0x2b03"
            },
            {
                "USB_path": "/4/3/3",
                "idProduct": "0x097e",
                "idVendor": "0x045e"
            },
            {
                "USB_path": "/4/3",
                "idProduct": "0x097b",
                "idVendor": "0x045e"
            },
            {
                "USB_path": "/4/2",
                "idProduct": "0xc52b",
                "idVendor": "0x046d"
            },
            {
                "USB_path": "/4",
                "idProduct": "0x5420",
                "idVendor": "0x0bda"
            },
            {
                "USB_path": "/3",
                "idProduct": "0x3549",
                "idVendor": "0x13d3"
            },
            {
                "USBMode": 2,
                "USB_path": "/2",
                "bDescriptorType": 1,
                "bDeviceProtocol": 0,
                "bLength": 18,
                "bMaxPacketSize0": 64,
                "bNumConfigurations": 1,
                "bcdDevice": "2.3",
                "bcdUSB": "2.0",
                "bcdUSBClass": 0,
                "bcdUSBSubClass": 0,
                "busNumber": 1,
                "device": "ZED mini MCU",
                "iManufacturer": 1,
                "iProduct": 2,
                "iSerial": 3,
                "idProduct": "0xf681",
                "idVendor": "0x2b03"
            }
        ],
        "USBMode": 3,
        "ZED Camera Module Detected": 63106,
        "ZED MCU Module Detected": 63105,
        "ZEDDetected": true,
        "error": [
            "<b>Low USB bandwidth</b> <br/> Read our FAQ to troubleshoot your USB connection issues. <a href='https://support.stereolabs.com/hc/en-us/articles/207635225' style='color: #d9ff42;' >Learn more</a><br/> <b>Frame drops:</b> 248/500<br/> <b>Frame tearing:</b> 11/500"
        ],
        "warning": [
            "<b>No GMSL driver found.</b><br/> USB camera are not impacted by this."
        ]
    },
    "Graphics Card": {
        "deviceCount": 1,
        "deviceDriverVersion": 12020,
        "devices": [
            {
                "arch": "Ampere",
                "computeCapability": "8.7",
                "cores": 2048,
                "name": "Orin",
                "totalMemoryBytes": "65893650432",
                "totalMemoryMB": 62841.08203125
            }
        ],
        "glx_info": "OpenGL vendor string: NVIDIA Corporation\nOpenGL renderer string: NVIDIA Tegra Orin (nvgpu)/integrated\nOpenGL core profile version string: 4.6.0 NVIDIA 540.3.0\nOpenGL core profile shading language version string: 4.60 NVIDIA\nOpenGL core profile context flags: (none)\nOpenGL core profile profile mask: core profile\nOpenGL core profile extensions:\nOpenGL version string: 4.6.0 NVIDIA 540.3.0\nOpenGL shading language version string: 4.60 NVIDIA\nOpenGL context flags: (none)\nOpenGL profile mask: (none)\nOpenGL extensions:\nOpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 540.3.0\nOpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20\nOpenGL ES profile extensions:\n",
        "initResult": 0,
        "valid": [
            "<b>Graphics card: </b> Orin"
        ]
    },
    "Processor": {
        "L4T version": "36.3.0",
        "OS": "Ubuntu 22.04.4 LTS ",
        "avxSupported": false,
        "coreCount": "12",
        "cpu": "",
        "datetime": "2024-08-27 10:27:07 AM",
        "hyperThreading": "0",
        "motherboard": "Jetson, NVIDIA",
        "ramCapacity": "65.893650",
        "ramUsage": "",
        "ramUsed": "7.427740",
        "threadCount": "12",
        "valid": [
            "<b>Processor: </b> ",
            "<b>Motherboard: </b> Jetson, NVIDIA"
        ]
    },
    "ZED SDK": {
        "CUDA Toolkit version": "V12.2.140 -->located in /usr/local/cuda-12.2/bin/nvcc",
        "CUDA loading": false,
        "GPU Driver Version": "Driver Version                            : N/A",
        "GPU Performance": "    Performance State                     : N/A",
        "ZED SDK Version (Diag)": "4.1.3",
        "ZED SDK Version (RT)": "4.1.3",
        "ai": {
            "checkAI": true,
            "cudaVersion": 12020,
            "cudnnCudartVersion": 12020,
            "cudnnVersion": 8904,
            "cudnnVersionExpected": 8904,
            "tensorVersion": 8602
        },
        "binFiles": [
            "libsl_ai.so",
            "libsl_zed.so",
            "libsl_zed_static.a"
        ],
        "resourcesFiles": [
            "neural_depth_3.6.model",
            "objects_accurate_3.2.model",
            "objects_medium_3.2.model",
            "objects_performance_3.2.model",
            "person_head_accurate_2.4.model",
            "person_head_performance_2.4.model",
            "person_reid_1.4.model",
            "positional_tracking_2.2.model",
            "skeleton_body18_3.2.model",
            "skeleton_body38_3.5.model"
        ],
        "resourcesFilesExpected": [
            "objects_performance_3.2"
        ],
        "valid": [
            "<b>ZED SDK version:</b> 4.1.3",
            "<b>CUDA version:</b> V12.2.140"
        ]
    }
}

If the Azure Kinect is idle (but still plugged in)

Summary
{
    "AI Models": {
        "info": [
            "MULTI CLASS DETECTION is not optimized",
            "MULTI CLASS MEDIUM DETECTION is not optimized",
            "MULTI CLASS ACCURATE DETECTION is not optimized",
            "HUMAN BODY FAST DETECTION is not optimized",
            "HUMAN BODY MEDIUM DETECTION is not optimized",
            "HUMAN BODY ACCURATE DETECTION is not optimized",
            "HUMAN BODY 38 FAST DETECTION is not optimized",
            "HUMAN BODY 38 MEDIUM DETECTION is not optimized",
            "HUMAN BODY 38 ACCURATE DETECTION is not optimized",
            "PERSON HEAD DETECTION is not optimized",
            "PERSON HEAD ACCURATE DETECTION is not optimized",
            "REID ASSOCIATION is not optimized",
            "NEURAL DEPTH is not optimized",
            "NEURAL PLUS DEPTH is not optimized"
        ]
    },
    "Camera Test": {
        "InternalDevicesCount": 2,
        "ZEDCount": 1,
        "camera": {
            "Device ID": 0,
            "Firmware": 1523,
            "Init Output": "OK",
            "Model": "ZED Mini",
            "Serial Number": 10027879,
            "valid": [
                "<b>Camera: </b>ZED Mini",
                "<b>Firmware: </b>1523"
            ]
        },
        "resolutions": {
            "HD1080@15": {
                "fps (input)": 15,
                "fps (output)": 15,
                "height (input)": 1080,
                "height (output)": 1080,
                "initialization": "OK",
                "status": true,
                "width (input)": 1920,
                "width (output)": 1920
            },
            "HD1080@30": {
                "fps (input)": 30,
                "fps (output)": 30,
                "height (input)": 1080,
                "height (output)": 1080,
                "initialization": "OK",
                "status": true,
                "width (input)": 1920,
                "width (output)": 1920
            },
            "HD2K@15": {
                "fps (input)": 15,
                "fps (output)": 15,
                "height (input)": 1242,
                "height (output)": 1242,
                "initialization": "OK",
                "status": true,
                "width (input)": 2208,
                "width (output)": 2208
            },
            "HD720@15": {
                "fps (input)": 15,
                "fps (output)": 15,
                "height (input)": 720,
                "height (output)": 720,
                "initialization": "OK",
                "status": true,
                "width (input)": 1280,
                "width (output)": 1280
            },
            "HD720@30": {
                "fps (input)": 30,
                "fps (output)": 30,
                "height (input)": 720,
                "height (output)": 720,
                "initialization": "OK",
                "status": true,
                "width (input)": 1280,
                "width (output)": 1280
            },
            "HD720@60": {
                "dropCount": -1,
                "fps (input)": 60,
                "fps (output)": 60,
                "height (input)": 720,
                "height (output)": 720,
                "imageCount": 502,
                "initialization": "OK",
                "status": true,
                "tearingCount": 0,
                "width (input)": 1280,
                "width (output)": 1280
            },
            "VGA@100": {
                "fps (input)": 100,
                "fps (output)": 100,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            },
            "VGA@15": {
                "fps (input)": 15,
                "fps (output)": 15,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            },
            "VGA@30": {
                "fps (input)": 30,
                "fps (output)": 30,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            },
            "VGA@60": {
                "fps (input)": 60,
                "fps (output)": 60,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            }
        },
        "sensors": {
            "Firmware": 515,
            "Init Output": "OK",
            "Model": "ZED Mini (MCU)",
            "Serial Number": 10027879
        }
    },
    "Devices": {
        "CorruptedFirmware": false,
        "GMSL driver": "",
        "GMSL driver compatiblity": "Unknown",
        "GMSLList": [
        ],
        "MCUDetected": true,
        "USBList": [
            {
                "USB_path": "/3/3/2",
                "idProduct": "0x097c",
                "idVendor": "0x045e"
            },
            {
                "USB_path": "/3/3/1",
                "idProduct": "0x097d",
                "idVendor": "0x045e"
            },
            {
                "USB_path": "/3/3",
                "idProduct": "0x097a",
                "idVendor": "0x045e"
            },
            {
                "USB_path": "/3",
                "idProduct": "0x0420",
                "idVendor": "0x0bda"
            },
            {
                "USBMode": 3,
                "USB_path": "/1",
                "bDescriptorType": 1,
                "bDeviceProtocol": 1,
                "bLength": 18,
                "bMaxPacketSize0": 9,
                "bNumConfigurations": 1,
                "bcdDevice": "1.0",
                "bcdUSB": "3.0",
                "bcdUSBClass": 239,
                "bcdUSBSubClass": 2,
                "busNumber": 2,
                "device": "ZED mini",
                "iManufacturer": 1,
                "iProduct": 2,
                "iSerial": 0,
                "idProduct": "0xf682",
                "idVendor": "0x2b03"
            },
            {
                "USB_path": "/4/3/3",
                "idProduct": "0x097e",
                "idVendor": "0x045e"
            },
            {
                "USB_path": "/4/3",
                "idProduct": "0x097b",
                "idVendor": "0x045e"
            },
            {
                "USB_path": "/4/2",
                "idProduct": "0xc52b",
                "idVendor": "0x046d"
            },
            {
                "USB_path": "/4",
                "idProduct": "0x5420",
                "idVendor": "0x0bda"
            },
            {
                "USB_path": "/3",
                "idProduct": "0x3549",
                "idVendor": "0x13d3"
            },
            {
                "USBMode": 2,
                "USB_path": "/2",
                "bDescriptorType": 1,
                "bDeviceProtocol": 0,
                "bLength": 18,
                "bMaxPacketSize0": 64,
                "bNumConfigurations": 1,
                "bcdDevice": "2.3",
                "bcdUSB": "2.0",
                "bcdUSBClass": 0,
                "bcdUSBSubClass": 0,
                "busNumber": 1,
                "device": "ZED mini MCU",
                "iManufacturer": 1,
                "iProduct": 2,
                "iSerial": 3,
                "idProduct": "0xf681",
                "idVendor": "0x2b03"
            }
        ],
        "USBMode": 3,
        "ZED Camera Module Detected": 63106,
        "ZED MCU Module Detected": 63105,
        "ZEDDetected": true,
        "valid": [
            "<b>USB Bandwidth: </b> OK"
        ],
        "warning": [
            "<b>No GMSL driver found.</b><br/> USB camera are not impacted by this."
        ]
    },
    "Graphics Card": {
        "deviceCount": 1,
        "deviceDriverVersion": 12020,
        "devices": [
            {
                "arch": "Ampere",
                "computeCapability": "8.7",
                "cores": 2048,
                "name": "Orin",
                "totalMemoryBytes": "65893650432",
                "totalMemoryMB": 62841.08203125
            }
        ],
        "glx_info": "OpenGL vendor string: NVIDIA Corporation\nOpenGL renderer string: NVIDIA Tegra Orin (nvgpu)/integrated\nOpenGL core profile version string: 4.6.0 NVIDIA 540.3.0\nOpenGL core profile shading language version string: 4.60 NVIDIA\nOpenGL core profile context flags: (none)\nOpenGL core profile profile mask: core profile\nOpenGL core profile extensions:\nOpenGL version string: 4.6.0 NVIDIA 540.3.0\nOpenGL shading language version string: 4.60 NVIDIA\nOpenGL context flags: (none)\nOpenGL profile mask: (none)\nOpenGL extensions:\nOpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 540.3.0\nOpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20\nOpenGL ES profile extensions:\n",
        "initResult": 0,
        "valid": [
            "<b>Graphics card: </b> Orin"
        ]
    },
    "Processor": {
        "L4T version": "36.3.0",
        "OS": "Ubuntu 22.04.4 LTS ",
        "avxSupported": false,
        "coreCount": "12",
        "cpu": "",
        "datetime": "2024-08-27 10:28:38 AM",
        "hyperThreading": "0",
        "motherboard": "Jetson, NVIDIA",
        "ramCapacity": "65.893650",
        "ramUsage": "",
        "ramUsed": "7.105593",
        "threadCount": "12",
        "valid": [
            "<b>Processor: </b> ",
            "<b>Motherboard: </b> Jetson, NVIDIA"
        ]
    },
    "ZED SDK": {
        "CUDA Toolkit version": "V12.2.140 -->located in /usr/local/cuda-12.2/bin/nvcc",
        "CUDA loading": false,
        "GPU Driver Version": "Driver Version                            : N/A",
        "GPU Performance": "    Performance State                     : N/A",
        "ZED SDK Version (Diag)": "4.1.3",
        "ZED SDK Version (RT)": "4.1.3",
        "ai": {
            "checkAI": true,
            "cudaVersion": 12020,
            "cudnnCudartVersion": 12020,
            "cudnnVersion": 8904,
            "cudnnVersionExpected": 8904,
            "tensorVersion": 8602
        },
        "binFiles": [
            "libsl_ai.so",
            "libsl_zed.so",
            "libsl_zed_static.a"
        ],
        "resourcesFiles": [
            "neural_depth_3.6.model",
            "objects_accurate_3.2.model",
            "objects_medium_3.2.model",
            "objects_performance_3.2.model",
            "person_head_accurate_2.4.model",
            "person_head_performance_2.4.model",
            "person_reid_1.4.model",
            "positional_tracking_2.2.model",
            "skeleton_body18_3.2.model",
            "skeleton_body38_3.5.model"
        ],
        "resourcesFilesExpected": [
            "objects_performance_3.2"
        ],
        "valid": [
            "<b>ZED SDK version:</b> 4.1.3",
            "<b>CUDA version:</b> V12.2.140"
        ]
    }
}

There are no changes to the dmesg nor lsusb - from a linux point of view, everything seems normal. The CPU load and RAM usage are fairly low, so I do not suspect a CPU issue.

Hi @wegr
Welcome to the Stereolabs community.

Are you using a PCIe adapter? What model?

Please note that all the USB3 ports of the Jetson Orin AGX devkit are connected to the same USB3 controller, so all the ports share the same channel.

Your problem can be caused by both powering and bandwidth issues.

Hello,

Thank you for answering - we are not using a PCIe Card. According to the Nvidia Orin SoC Technical Reference Manual on Page 7021, Nvidia says:

Note that due to throughput limitations in the xHCI controller, USB 3.2 port 0 and port 1 are connected to the same SuperSpeedPlus hub to share 10 Gbps total bandwidth, while USB 3.2 port 2 and port 3 share another 10 Gbps total bandwidth. Note also that in the case of two SuperSpeed devices connected to the same SuperSpeedPlus hub, due to scheduling policy reasons each SuperSpeed device may not be able sustain the full SuperSpeed unit bandwidth if the other is device is active.

I follow this policy by attaching the two cameras to Port 1 and Port 3. Following this logic, the ZEDmini should have 10GBit/s of possible Bandwidth available.

As I suspect some scheduling-policy issue in the Orin-Controller as well, I also opened the topic in the Nvidia-Developer forums. But as the Azure Kinect works flawlessly, I also suspect some issue directly targeting the ZEDmini.

Are there some known “tuning screws” for the USB performance?

Thank you for your time :slight_smile:

Normally the PCIe additional board solves this kind of issue.
In any case, did you try to ingest additional 5V power to exclude powering issues?

I will try running it from a PCIe card - but that cannot be the solution for the final product, as we require the PCIe port for a Fiber-Optic Network Interface. I have to organize such an adapter card first.

I briefly tested running the ZEDmini off a powered USB-Hub - but that brought no improvement.

The USB 3 Complex should be rated at 20 Gbit/s - according to the Technical Reference Manual - but it somehow causes problems way less than that, which is rather confusing.

A user from the Nvidia-Forums suspects an issue with the ISR, that interrupts from the XUSB-Subsystem might get lost due to slow “memcpy” (or equivalent) calls.

Thank you for your time :slight_smile:

Can you send the output of the command lsusb -t -v with both the Kinect and ZED Mini connected?

Sure - I have now tested several combinations.

Both cameras connected to different “Main Ports” to the DevKit Also see: Image in original post.

$ lsusb -tv
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 10000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
    |__ Port 1: Dev 6, If 0, Class=Video, Driver=uvcvideo, 5000M
        ID 2b03:f682 STEREOLABS ZED-M camera
    |__ Port 1: Dev 6, If 1, Class=Video, Driver=uvcvideo, 5000M
        ID 2b03:f682 STEREOLABS ZED-M camera
    |__ Port 3: Dev 2, If 0, Class=Hub, Driver=hub/4p, 10000M
        ID 0bda:0420 Realtek Semiconductor Corp.
        |__ Port 3: Dev 3, If 0, Class=Hub, Driver=hub/2p, 5000M
            ID 045e:097a Microsoft Corp.
            |__ Port 1: Dev 4, If 2, Class=Vendor Specific Class, Driver=, 5000M
                ID 045e:097d Microsoft Corp.
            |__ Port 1: Dev 4, If 0, Class=Video, Driver=, 5000M
                ID 045e:097d Microsoft Corp.
            |__ Port 1: Dev 4, If 1, Class=Video, Driver=, 5000M
                ID 045e:097d Microsoft Corp.
            |__ Port 2: Dev 5, If 0, Class=Vendor Specific Class, Driver=, 5000M
                ID 045e:097c Microsoft Corp.
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
    |__ Port 2: Dev 8, If 0, Class=Human Interface Device, Driver=usbfs, 12M
        ID 2b03:f681 STEREOLABS ZED-M HID Interface
    |__ Port 3: Dev 3, If 0, Class=Wireless, Driver=rtk_btusb, 12M
        ID 13d3:3549 IMC Networks
    |__ Port 3: Dev 3, If 1, Class=Wireless, Driver=rtk_btusb, 12M
        ID 13d3:3549 IMC Networks
    |__ Port 4: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M
        ID 0bda:5420 Realtek Semiconductor Corp.
        |__ Port 3: Dev 6, If 0, Class=Hub, Driver=hub/4p, 480M
            ID 045e:097b Microsoft Corp.
            |__ Port 3: Dev 7, If 1, Class=Audio, Driver=snd-usb-audio, 480M
                ID 045e:097e Microsoft Corp.
            |__ Port 3: Dev 7, If 2, Class=Human Interface Device, Driver=usbhid, 480M
                ID 045e:097e Microsoft Corp.
            |__ Port 3: Dev 7, If 0, Class=Audio, Driver=snd-usb-audio, 480M
                ID 045e:097e Microsoft Corp.
        |__ Port 2: Dev 5, If 2, Class=Human Interface Device, Driver=usbhid, 12M
            ID 046d:c52b Logitech, Inc. Unifying Receiver
        |__ Port 2: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 12M
            ID 046d:c52b Logitech, Inc. Unifying Receiver
        |__ Port 2: Dev 5, If 1, Class=Human Interface Device, Driver=usbhid, 12M
            ID 046d:c52b Logitech, Inc. Unifying Receiver

Running ZEDmini from external USB-Hub

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 10000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
    |__ Port 1: Dev 16, If 0, Class=Hub, Driver=hub/3p, 10000M
        ID 0424:7206 Microchip Technology, Inc. (formerly SMSC) 
        |__ Port 3: Dev 18, If 0, Class=Hub, Driver=hub/5p, 10000M
            ID 0424:7206 Microchip Technology, Inc. (formerly SMSC) 
            |__ Port 2: Dev 19, If 0, Class=Video, Driver=uvcvideo, 5000M
                ID 2b03:f682 STEREOLABS ZED-M camera
            |__ Port 2: Dev 19, If 1, Class=Video, Driver=uvcvideo, 5000M
                ID 2b03:f682 STEREOLABS ZED-M camera
        |__ Port 1: Dev 17, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
            ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
    |__ Port 3: Dev 3, If 0, Class=Hub, Driver=hub/4p, 10000M
        ID 0bda:0420 Realtek Semiconductor Corp. 
        |__ Port 3: Dev 13, If 0, Class=Hub, Driver=hub/2p, 5000M
            ID 045e:097a Microsoft Corp. 
            |__ Port 1: Dev 15, If 2, Class=Vendor Specific Class, Driver=, 5000M
                ID 045e:097d Microsoft Corp. 
            |__ Port 1: Dev 15, If 0, Class=Video, Driver=uvcvideo, 5000M
                ID 045e:097d Microsoft Corp. 
            |__ Port 1: Dev 15, If 1, Class=Video, Driver=uvcvideo, 5000M
                ID 045e:097d Microsoft Corp. 
            |__ Port 2: Dev 14, If 0, Class=Vendor Specific Class, Driver=, 5000M
                ID 045e:097c Microsoft Corp. 
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
    |__ Port 2: Dev 19, If 0, Class=Hub, Driver=hub/4p, 480M
        ID 0424:4206 Microchip Technology, Inc. (formerly SMSC) 
        |__ Port 4: Dev 21, If 0, Class=Vendor Specific Class, Driver=, 480M
            ID 0424:7240 Microchip Technology, Inc. (formerly SMSC) 
        |__ Port 4: Dev 21, If 1, Class=, Driver=, 480M
            ID 0424:7240 Microchip Technology, Inc. (formerly SMSC) 
        |__ Port 3: Dev 20, If 0, Class=Hub, Driver=hub/6p, 480M
            ID 0424:4206 Microchip Technology, Inc. (formerly SMSC) 
            |__ Port 1: Dev 22, If 2, Class=Human Interface Device, Driver=usbhid, 12M
                ID 046d:c52b Logitech, Inc. Unifying Receiver
            |__ Port 1: Dev 22, If 0, Class=Human Interface Device, Driver=usbhid, 12M
                ID 046d:c52b Logitech, Inc. Unifying Receiver
            |__ Port 1: Dev 22, If 1, Class=Human Interface Device, Driver=usbhid, 12M
                ID 046d:c52b Logitech, Inc. Unifying Receiver
            |__ Port 6: Dev 24, If 0, Class=Vendor Specific Class, Driver=, 480M
                ID 0424:7260 Microchip Technology, Inc. (formerly SMSC) 
            |__ Port 6: Dev 24, If 1, Class=Human Interface Device, Driver=usbhid, 480M
                ID 0424:7260 Microchip Technology, Inc. (formerly SMSC) 
            |__ Port 2: Dev 23, If 0, Class=Human Interface Device, Driver=usbhid, 12M
                ID 2b03:f681 STEREOLABS ZED-M HID Interface
    |__ Port 3: Dev 3, If 0, Class=Wireless, Driver=rtk_btusb, 12M
        ID 13d3:3549 IMC Networks 
    |__ Port 3: Dev 3, If 1, Class=Wireless, Driver=rtk_btusb, 12M
        ID 13d3:3549 IMC Networks 
    |__ Port 4: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M
        ID 0bda:5420 Realtek Semiconductor Corp. 
        |__ Port 3: Dev 17, If 0, Class=Hub, Driver=hub/4p, 480M
            ID 045e:097b Microsoft Corp. 
            |__ Port 3: Dev 18, If 1, Class=Audio, Driver=snd-usb-audio, 480M
                ID 045e:097e Microsoft Corp. 
            |__ Port 3: Dev 18, If 2, Class=Human Interface Device, Driver=usbhid, 480M
                ID 045e:097e Microsoft Corp. 
            |__ Port 3: Dev 18, If 0, Class=Audio, Driver=snd-usb-audio, 480M
                ID 045e:097e Microsoft Corp. 
        |__ Port 2: Dev 6, If 2, Class=Human Interface Device, Driver=usbhid, 12M
            ID 046d:c52b Logitech, Inc. Unifying Receiver
        |__ Port 2: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 12M
            ID 046d:c52b Logitech, Inc. Unifying Receiver
        |__ Port 2: Dev 6, If 1, Class=Human Interface Device, Driver=usbhid, 12M
            ID 046d:c52b Logitech, Inc. Unifying Receiver

Both cameras connected to the same USB-Hub:

$ lsusb -tv
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 10000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
    |__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/3p, 10000M
        ID 0424:7206 Microchip Technology, Inc. (formerly SMSC) 
        |__ Port 3: Dev 4, If 0, Class=Hub, Driver=hub/5p, 10000M
            ID 0424:7206 Microchip Technology, Inc. (formerly SMSC) 
            |__ Port 2: Dev 9, If 1, Class=Video, Driver=uvcvideo, 5000M
                ID 2b03:f682 STEREOLABS ZED-M camera
            |__ Port 2: Dev 9, If 0, Class=Video, Driver=uvcvideo, 5000M
                ID 2b03:f682 STEREOLABS ZED-M camera
            |__ Port 5: Dev 10, If 0, Class=Hub, Driver=hub/2p, 10000M
                ID 045e:097a Microsoft Corp. 
                |__ Port 1: Dev 12, If 1, Class=Video, Driver=, 5000M
                    ID 045e:097d Microsoft Corp. 
                |__ Port 1: Dev 12, If 2, Class=Vendor Specific Class, Driver=, 5000M
                    ID 045e:097d Microsoft Corp. 
                |__ Port 1: Dev 12, If 0, Class=Video, Driver=, 5000M
                    ID 045e:097d Microsoft Corp. 
                |__ Port 2: Dev 11, If 0, Class=Vendor Specific Class, Driver=, 5000M
                    ID 045e:097c Microsoft Corp. 
        |__ Port 1: Dev 5, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
            ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
    |__ Port 3: Dev 3, If 0, Class=Hub, Driver=hub/4p, 10000M
        ID 0bda:0420 Realtek Semiconductor Corp. 
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        ID 0424:4206 Microchip Technology, Inc. (formerly SMSC) 
        |__ Port 3: Dev 7, If 0, Class=Hub, Driver=hub/6p, 480M
            ID 0424:4206 Microchip Technology, Inc. (formerly SMSC) 
            |__ Port 1: Dev 11, If 1, Class=Human Interface Device, Driver=usbhid, 12M
                ID 046d:c52b Logitech, Inc. Unifying Receiver
            |__ Port 1: Dev 11, If 2, Class=Human Interface Device, Driver=usbhid, 12M
                ID 046d:c52b Logitech, Inc. Unifying Receiver
            |__ Port 1: Dev 11, If 0, Class=Human Interface Device, Driver=usbhid, 12M
                ID 046d:c52b Logitech, Inc. Unifying Receiver
            |__ Port 6: Dev 13, If 1, Class=Human Interface Device, Driver=usbhid, 480M
                ID 0424:7260 Microchip Technology, Inc. (formerly SMSC) 
            |__ Port 6: Dev 13, If 0, Class=Vendor Specific Class, Driver=, 480M
                ID 0424:7260 Microchip Technology, Inc. (formerly SMSC) 
            |__ Port 2: Dev 14, If 0, Class=Human Interface Device, Driver=, 12M
                ID 2b03:f681 STEREOLABS ZED-M HID Interface
            |__ Port 5: Dev 15, If 0, Class=Hub, Driver=hub/4p, 480M
                ID 045e:097b Microsoft Corp. 
                |__ Port 3: Dev 16, If 2, Class=Human Interface Device, Driver=usbhid, 480M
                    ID 045e:097e Microsoft Corp. 
                |__ Port 3: Dev 16, If 0, Class=Audio, Driver=snd-usb-audio, 480M
                    ID 045e:097e Microsoft Corp. 
                |__ Port 3: Dev 16, If 1, Class=Audio, Driver=snd-usb-audio, 480M
                    ID 045e:097e Microsoft Corp. 
        |__ Port 4: Dev 9, If 0, Class=Vendor Specific Class, Driver=, 480M
            ID 0424:7240 Microchip Technology, Inc. (formerly SMSC) 
        |__ Port 4: Dev 9, If 1, Class=, Driver=, 480M
            ID 0424:7240 Microchip Technology, Inc. (formerly SMSC) 
    |__ Port 3: Dev 3, If 0, Class=Wireless, Driver=rtk_btusb, 12M
        ID 13d3:3549 IMC Networks 
    |__ Port 3: Dev 3, If 1, Class=Wireless, Driver=rtk_btusb, 12M
        ID 13d3:3549 IMC Networks 
    |__ Port 4: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M
        ID 0bda:5420 Realtek Semiconductor Corp. 
        |__ Port 2: Dev 6, If 2, Class=Human Interface Device, Driver=usbhid, 12M
            ID 046d:c52b Logitech, Inc. Unifying Receiver
        |__ Port 2: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 12M
            ID 046d:c52b Logitech, Inc. Unifying Receiver
        |__ Port 2: Dev 6, If 1, Class=Human Interface Device, Driver=usbhid, 12M
            ID 046d:c52b Logitech, Inc. Unifying Receiver

It’s always the same issue - each camera on its own runs fine. The ZEDmini starts to stutter once one of the two cameras runs at 1080p30 and fails completely when both should stream in 1080p30.
I wasn’t yet able to test a separate PCIe-USB-Controller.

Hi @wegr
in the first “tree” the Kinect is connected to the same bus as the STEREOLABS ZED-M camera (Bus 02), hence sharing the same bandwidth.
Please note that all the “port” of the same “bus” share the bandwidth.
What I can see here is that there is a single USB3 10000M bus (Bus 02) and a single USB2 480M bus (Bus 01)

The same happens for all the outputs: STEREOLABS ZED-M camera 5000M and Microsoft Corp. 5000M are connected to the same Bus 02, hence sharing the bandwidth.

On the Bus 01, which is USB2, there are the STEREOLABS ZED-M HID Interface and the HID and audio peripherals of the Kinect.

This confirms my hypothesis that the ZED Mini is not working because there is not enough bandwidth to handle both devices simultaneously.

Hello Myzhar,

On Nvidia Orin, Bus 01 is the USB 2.0 Controller (4x480Mbit/s), while Bus 02 is the USB 3.2 Gen 2 (2x 10GBit/s) Controller - at least, according to the Nvidia Documentation.
The 2x 10GBit/s USB 3.2 Gen 2 Controller is split into 4 Ports, of which Ports 0 and 1 share 10 GBit/s and Ports 2 and 3 share another 10 GBit/s. (Due to weird enumeration, these ports are sometimes numbered 0 to 3 or 1 to 4).

At least, that’s what’s documented. Nvidia is not very clear about the true maximum throughput - unfortunately.

I now combined the ZEDmini with a Logitech FullHD webcam - with which the ZED works fine. But the maximum throughput I get out of that Webcam is unfortunately limited to FullHD 30 at 4:2:2 chroma subsampling. (So max Throughput is at about 1Gbit/s).

As the experienced outcomes and the documented things differ, I try to dig deeper and let you know if I find out something.
I only have one ZEDmini at the desk - as you at Stereolabs also work with AGX Orins (e.g. for the ZedBox), have you ever tried connecting two ZED minis to it?

Best regards

Yes, normally we notice issues when connecting more than two cameras, that’s why I have doubts concerning the availability of two 10 GBit/s channels.

Hello Myzhar,

linuxdev from the Nvidia-Forums recommended looking at the interrupt counter. The findings seem to support the thesis of a plain bandwidth limitation in the USB-Complex.
I thought, I’d share the results of my tests in here. I measured the system interrupt count coming from the USB-Complex and the total (systemwide) rescheduled interrupts. (Only CPU0 on Orin is able to handle IRQ requests - as the other cores are simply not connected. IRQs scheduled to other cores need to get rescheduled to CPU0 by the linux scheduler)

Test USB-Interrupts per second Total (systemwide) rescheduled interrupts per second
Nothing running 4.7 4.4
ZEDmini @ 1080p 8441.1 6.5
ZEDmini @ 720p 4222.7 4.23
Azure Kinect @ 1080p 1173.7 9.56
Azure Kinect @ 720p 881.4 4.9
Both (2x 720p) 4258.5 102.3
Both (Azure@720p, ZEDmini@1080p) 4850.2 13.1
Both (Azure@1080p, ZEDmini@720p) 5369.2 6.22

From the solo-camera-measurements, we would expect roughly 10’000 interrupts per second, when running the ZEDmini at 1080p while running the AzureKinect at 720p. Lacking further insight into the system, I guess, the USB-Complex is simply overloaded in this case and (as a side-effect of deleted packages) generates less interrupts than expected.

From this, I would conclude, that the Nvidia-Documentation is wrong on that topic - or am I missing something?

Thank you for your time :slight_smile:

I think the documentation is unclear because, at first reading, it suggests a total bandwidth of 2x 10 Gbit/s, which is not the case because you have a single controller and two internal hubs, hence 10 Gbit/s shared between them.