[ROS2 Wrapper] 4 ZED2i cameras same device in docker container

Hi,

I have 4 ZED2i connected on the device via USB3.0 separate controllers, but when i`m trying to launch them i see that only 2 got recognized. If i remove serial numbers from launch file, i steal see only 2 cameras, but with that appears warning that says LOW USB BANDWIDTH. Can you help me to connect all 4 of them?

Hi all! I’ll add some info to previous post.
So, i’ve already asked about usb bandwidth here, and now we have 2 different PCIe expansion cards. Each has 2 ZED2i cameras, so, as i understand - it shouldnt be usn brandwidth issue
We tried to run in different modes (VGA@15, 720@15), but all 4 are not running at the same time.
If we have serial numbers in launch file - we see log1.txt, if we remove them - we can see log2.txt
log2.txt (118.6 KB)
log1.txt (94.3 KB)
Moreover, when I just access cameras via python - all cameras are available
But as soon as we start testing inside ros&docker, they turn off one by one (python sdk does not see them, but “v4l2-ctl --list-devices” show them

Hi @pixml and @otstanteplz
do the camera connect if you try to start the 4 nodes separately?
We do not suggest connecting 2 cameras to the same USB3 channel? Are the 4 cameras connected to 4 different channels?
Can you post the output of the command $ lsusb -t -v?

Thanks for quick response!
About 4 nodes separately @otstanteplz will answer tomorrow.
If I understand you correctly, usb3 channel is equal to usb controller, so we have 2 PCI Expansion cards == we have 2 controllers/channels
First PCI Expansion card connected with 1 and 2 Zed2i cameras, and second connected to 3 and 4 Zed2i cameras
We understand that it would be better to have a controller for each camera, but in the previously we discussed that the 2 cameras per expansion card should work…
Output of lsusb -t -v

/: Bus 10.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
ID 1d6b:0003 Linux Foundation 3.0 root hub
|__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/1p, 5000M
ID 2109:0211 VIA Labs, Inc.
|__ Port 1: Dev 4, If 0, Class=Hub, Driver=hub/1p, 5000M
ID 2109:0211 VIA Labs, Inc.
|__ Port 1: Dev 6, If 1, Class=Video, Driver=uvcvideo, 5000M
ID 2b03:f880
|__ Port 1: Dev 6, If 0, Class=Video, Driver=uvcvideo, 5000M
ID 2b03:f880
|__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/1p, 5000M
ID 2109:0211 VIA Labs, Inc.
|__ Port 1: Dev 5, If 0, Class=Video, Driver=uvcvideo, 5000M
ID 2b03:f880
|__ Port 1: Dev 5, If 1, Class=Video, Driver=uvcvideo, 5000M
ID 2b03:f880
/: Bus 09.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
ID 1d6b:0002 Linux Foundation 2.0 root hub
|__ Port 1: Dev 2, If 0, Class=Video, Driver=uvcvideo, 480M
ID 0ac8:3500 Z-Star Microelectronics Corp.
|__ Port 1: Dev 2, If 1, Class=Video, Driver=uvcvideo, 480M
ID 0ac8:3500 Z-Star Microelectronics Corp.
|__ Port 1: Dev 2, If 2, Class=Audio, Driver=snd-usb-audio, 480M
ID 0ac8:3500 Z-Star Microelectronics Corp.
|__ Port 1: Dev 2, If 3, Class=Audio, Driver=snd-usb-audio, 480M
ID 0ac8:3500 Z-Star Microelectronics Corp.
|__ Port 2: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
ID 2109:2211 VIA Labs, Inc.
|__ Port 1: Dev 8, If 0, Class=Hub, Driver=hub/4p, 480M
ID 2109:2211 VIA Labs, Inc.
|__ Port 1: Dev 9, If 0, Class=Hub, Driver=hub/2p, 480M
ID 0424:2512 Microchip Technology, Inc. (formerly SMSC) USB 2.0 Hub
|__ Port 2: Dev 11, If 0, Class=Human Interface Device, Driver=, 12M
ID 2b03:f881
|__ Port 4: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M
ID 2109:2211 VIA Labs, Inc.
|__ Port 1: Dev 5, If 0, Class=Hub, Driver=hub/2p, 480M
ID 0424:2512 Microchip Technology, Inc. (formerly SMSC) USB 2.0 Hub
|__ Port 2: Dev 7, If 0, Class=Human Interface Device, Driver=, 12M
ID 2b03:f881
/: Bus 08.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
ID 1d6b:0003 Linux Foundation 3.0 root hub
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/1p, 5000M
ID 2109:0211 VIA Labs, Inc.
|__ Port 1: Dev 4, If 0, Class=Video, Driver=uvcvideo, 5000M
ID 2b03:f880
|__ Port 1: Dev 4, If 1, Class=Video, Driver=uvcvideo, 5000M
ID 2b03:f880
|__ Port 3: Dev 3, If 0, Class=Hub, Driver=hub/1p, 5000M
ID 2109:0211 VIA Labs, Inc.
|__ Port 1: Dev 5, If 0, Class=Video, Driver=uvcvideo, 5000M
ID 2b03:f880
|__ Port 1: Dev 5, If 1, Class=Video, Driver=uvcvideo, 5000M
ID 2b03:f880
/: Bus 07.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
ID 1d6b:0002 Linux Foundation 2.0 root hub
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
ID 2109:3431 VIA Labs, Inc. Hub
|__ Port 1: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
ID 2109:2211 VIA Labs, Inc.
|__ Port 1: Dev 5, If 0, Class=Hub, Driver=hub/2p, 480M
ID 0424:2512 Microchip Technology, Inc. (formerly SMSC) USB 2.0 Hub
|__ Port 2: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 12M
ID 2b03:f881
|__ Port 3: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M
ID 2109:2211 VIA Labs, Inc.
|__ Port 1: Dev 7, If 0, Class=Hub, Driver=hub/2p, 480M
ID 0424:2512 Microchip Technology, Inc. (formerly SMSC) USB 2.0 Hub
|__ Port 2: Dev 8, If 0, Class=Human Interface Device, Driver=, 12M
ID 2b03:f881
/: Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
ID 1d6b:0003 Linux Foundation 3.0 root hub
/: Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
ID 1d6b:0002 Linux Foundation 2.0 root hub
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 5000M
ID 1d6b:0003 Linux Foundation 3.0 root hub
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M
ID 1d6b:0002 Linux Foundation 2.0 root hub
|__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
ID 062a:4101 MosArt Semiconductor Corp. Wireless Keyboard/Mouse
|__ Port 1: Dev 2, If 1, Class=Human Interface Device, Driver=usbhid, 12M
ID 062a:4101 MosArt Semiconductor Corp. Wireless Keyboard/Mouse
|__ Port 8: Dev 3, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M
ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port
|__ Port 10: Dev 4, If 0, Class=Mass Storage, Driver=ums-realtek, 480M
ID 0bda:0153 Realtek Semiconductor Corp. 3-in-1 (SD/SDHC/SDXC) Card Reader
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
ID 1d6b:0002 Linux Foundation 2.0 root hub
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
ID 8087:8000 Intel Corp.
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
ID 1d6b:0002 Linux Foundation 2.0 root hub
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
ID 8087:8008 Intel Corp.

Hi, I tried run them separately, launch order doesn’t matter, it always one camera with same serial number that doesn’t work. I tried run this serial number first, but result is still the same

According to the device tree the 4 cameras are all connected to the Bus 10, one on the Port 4, the remaining 3 to the Port 1.
Are you using hubs?

Well, I’m not very good at hardware, but as i understand:

  1. 4 cameras are all connected to the Bus 10, but two of them connected to the Port4, two of them connected to the Port2
  2. We don’t use usb hubs, we use 2 PCI Expansion cards (this is first, second is the same, but with 3.0 ports)

I think i was wrong a bit
I see same tree on bus 8:

Yes, this seems correct, the problem is that 2x ZED 2i connected to the same bus channel can conflict.

Have you tried this configuration outside the docker container?

So, i’m not sure, but it looks like this is not about ZED cameras, because lsusb -d 2b03: -v shows another info
log_lsusb2b03.txt (73.5 KB)

And one more interesting thing
If we pull out the unworked zed camera from usb port and then plug it back in same usb port, this zed camera can work, but then any other zed2i camera can turn off (even from another bus)

The ZED 2i when connected to the USB port creates two devices:

  • 2b03:f880 → UVC video module (USB3)
  • 2b03:f881 → HID sensors data module (USB2)

The fact that when you connect one camera another one stops to work make me think that this is also a power issue.

@adujardin already suggested to be sure that the expansion cards are correctly powered. Is the power configuration correct?

1 Like

We don’t use ROS inside docker, but i think we can try
I will describe what a typical situation looks like:

  1. We connected 4 cameras
  2. We checked that we see them through Zed Explorer and python sl.Camera.get_device_list()
  3. We are starting to test our ROS pipeline
  4. In ROS only two or three cameras turn on
  5. We stop ROS, look at the cameras through Zed Explorer or python and see only 2 or 3 cameras
  6. Loop repeat

Yes, the cards must be powered, but I will ask our engineers to check again

@pixml the title of this thread is 4 ZED2i cameras same device in docker container… please change it if it’s not correct

Oh, my mistake, sorry
I meant we don’t use ROS outside of docker, but we can try (as you suggested)

Upd I tried to run zed cameras separetly via default launch file and than run my own launch file which starts other nodes and this worked out, but when i trying to run all zeds via one launch file i’m facing issue with one camera missing.
I’m attaching my launch file where i trying to launch all 4 cameras
i don’t have enough permissions to attach files, so here link to gist multiple_zed2i.launch.py · GitHub

Can you try to remove this line?

The camera id is assigned by the OS, if camera id and serial number do not match you can have issues.

Thank you, it worked like a charm
Upd @Myzhar It workerd one time, when i tried launch second time after a while, the issue came back
ZED Explorer stopped recognizing camera and suggesting to plug cam to another usb port, it goes away after turning off the power from this cam

It is possible that for power issues the cameras are excluded by the USB3 controller.
You can recover them by using the sl::reboot command with the S/N as the parameter.

A tip that you could use is to add a little delay (e.g. 1 second) between the different camera openings.

A curiosity, have you tried the launch file of our multi-camera tutorial?

No, i have not tried yours launch, but i defently will and let you know the results

@Myzhar I tried your launch and the problem persists. But the reboot command helped, the only thing is that I can’t run it inside the container, i always getting FAILURE, but outside it works fine. The main idea, that i will reboot all my serials and then launch all nodes (it works if i reboot them outside of container). Can you help me with that? i tried to look in source code to understand why reboot returns FAILURE but cant find anything except this part zed-python-api/sl_c.pxd at master · stereolabs/zed-python-api · GitHub