Very difficult to open multiple ZED2 cameras (multiprocess or in sequence)

Having difficulty connecting four ZED2 cameras to a Windows 11 PC.

When using one camera at a time or when using the ZED explorer, all cameras open under 10sec. Once in a while it won’t start and require reboot / re-plugging, but overall it is usable.

However it gets more and more difficult adding new cameras.

I wrote a simple script to benchmark this:

import time
import pyzed.sl as sl

def get_connected(echo=False):
    connected = sl.Camera.get_device_list()
    camera_serials = []
    for device in connected:
        camera_serials.append(device.serial_number)
        if echo:
            ic(
                device.camera_state,
                device.id,
                device.path,
                device.camera_model,
                device.serial_number,
                device.input_type
            )
            time.sleep(1)
    return camera_serials

def main():
    camera_serials = get_connected()

    test_start_time = time.time()
    print('--- test camera one by one ---')
    for sn in camera_serials:
        start_time = time.time()
        init_params = sl.InitParameters(
            camera_resolution = sl.RESOLUTION.VGA,
		  	camera_fps = 15,
            open_timeout_sec = 5.,
        )
        init_params.set_from_serial_number(sn)
        camera = sl.Camera()
        status = camera.open(init_params)
        elapsed = round(time.time() - start_time, 1)
        print(f'{sn}: open {status} in { elapsed }s')
        time.sleep(1)
        status = camera.close()
        print(f'{sn}: close {status}')
        time.sleep(1)
    
    print('--- batch open ---')
    open_cameras = {}
    for sn in camera_serials:
        start_time = time.time()
        init_params = sl.InitParameters(
            camera_resolution = sl.RESOLUTION.VGA,
		  	camera_fps = 15,
            open_timeout_sec = 5.,
        )
        init_params.set_from_serial_number(sn)
        camera = sl.Camera()
        open_cameras[sn] = camera
        status = camera.open(init_params)
        elapsed = round(time.time() - start_time, 1)
        print(f'{sn}: open {status} in { elapsed }s')
        time.sleep(1)

    print('--- batch close ---')
    for sn in camera_serials:
        camera = open_cameras[sn]
        status = camera.close()
        print(f'{sn}: close {status}')
        time.sleep(1)

    print(f'--- test completed in { time.time() - test_start_time }s ---')

if __name__ == "__main__":
    main()

When we’re lucky we get:

--- test camera one by one ---
20832444: open SUCCESS in 9.9s
20832444: close None
20513324: open SUCCESS in 9.6s
20513324: close None
23551088: open SUCCESS in 9.5s
23551088: close None
22198948: open SUCCESS in 9.3s
22198948: close None

--- batch open ---
20832444: open SUCCESS in 9.4s
20513324: open SUCCESS in 11.1s
23551088: open SUCCESS in 413.2s
22198948: open SUCCESS in 238.3s

More often it just hangs forever (or maybe we haven’t waited long enough), or receive CAMERA NOT DETECTED after several minutes. Occasionally ctrl-c won’t quit the app and we have to force quit.

In the real app (which use multi-processing) the situation is much worse. When we are very lucky it takes 10-30mins to start; Sometime it says CAMERA NOT DETECTED after a long wait; Most of the time we cannot tell if it has hanged and decide to quit.

We are using MSI PRO Z790-P WIFI, Intel 13900K, rtx4090, and tried a few PCIe USB extension boards. Tried connecting less cameras (2-3) as well as experimenting with different ports (only motherboard / motherboard + PCIe / only PCIe / all USB3 / USB3+2 / powered hubs / another PC), but nothing really help. With 2 cameras it is ~70% success rate; with 3 cameras less than 30%; with 4 cameras we usually just give up after 15mins.

The PC is running windows 11 Home, cuda 12.1, ZED SDK 4.0.7, Python 3.10.13 (Anaconda)

Any help is much appreciated!

Hi,

This is indeed not the expected behaviour, the camera should not be that slow to open.

We are currently investigating on our side and we hope we can push a fix regarding this issue in the coming release (v4.0.8) of the ZED SDK.

I’ll keep you updated on this post when I have some news.

Thanks for the report.

Best regards,

Stereolabs Support

Hi @BenjaminV are there an ETA for the new SDK? And were there similar reports before and have they find a solution?

FYI we tests again today and here’s the result (considered very stable comparing with previous tests)

1st test(4 cam)
Successful
Duration: ~15 mins

2nd test(4 cam)
Successful
Duration: ~1min

3rd test(4 cam)
Failed with 3 cam
Duration: ~3 mins

— Unplug and replug —

4th test(4 cam)
Successful
Duration: ~1 min

5th test(4 cam)
Successful
Duration: ~20 min

6th test(4 cam)
Successful
Duration: ~5 mins

7th test(4 cam)
Successful
Duration: ~1 min

8th test(4 cam)
Successful
Duration: ~15 min

9th test(4 cam)
Successful
Duration: ~10 min

10th test(4 cam)
Successful
Duration: ~8 min

11th test(4 cam)
Failed with 3 cam
Duration: ~13 mins

— Unplug and replug —

12th test(4 cam)
Successful
Duration: ~2 mins

13th test(4 cam)
Successful
Duration: ~12 mins

14th test(4 cam)
Successful
Duration: ~2 mins

15th test(4 cam)
Successful
Duration: ~ 12mins

I’ve had similar issues on Windows 10; which I’m still having, but one thing I found that seemed to somewhat help with stability was running the cameras through one (or more, depending on USB bandwidth needs) powered USB 3.0 hubs, with short USB cables (I can confirm 3 cameras work fine at HD1080 through an Amazon Basics 7-port USB 3 hub – no longer available – and these ones are working fine too). Another setup that also seemed to be more stable, if you need long USB cables, is using USB cables that have power injection at the end rather than a powered hub.

I don’t actually know if it’s a power problem, our attempts to solve this issue were all pretty hand-wavey, but getting external power into the USB line close to the cameras did seem to help a little.

I don’t know if this will help you at all. Like I said it hasn’t completely solved the issue but the system does seem to be a bit more stable.

Fwiw, on Ubuntu, I’ve never seen the issue.

@JC3 thanks for sharing! We tried putting a powered hub before each cameras (i.e. 4 powered hub for 4 cameras) but can’t say if it really help. It would be hard-pressed to believe it is power problem, given that it would happen even with direct connection. I suppose I will try Ubuntu and see if it will fix the issue. May I know which version(s) of Ubuntu did you use? Are those native install or WSL/2?

@BenjaminV will 4.0.8 be released in the coming week? Would like to test that before attempting with Ubuntu…

Ubuntu 22, native install.

Hi,

Unfortunately we will not have a fix before the release date (which is this week). However we reproduce this issue thanks to your script and are actively working on a fix. Even so we have improvements already implemented, it’s low-level that needs a strong QA before being shipped because we need to be absolutely sure that the reliability has not decreased on all the platforms.

Thank you all for your patience, and sorry for the delay. If one of you is interested, we can ship preview previews before the fix will actually be released.

Good news, the fix is done and will be sneaked into the patch that will be released this week :partying_face:

2 Likes

Thanks, looking forward to the release!

Can confirm, Windows 10 device initialization with 4.0.8 is noticeably snappier than before.

1 Like