ZED SDK 4.1 problems in jetson and docker

NVG97 · April 19, 2024, 8:59pm

Hi!

I have the next script for testing the image capture and video recording of the zed camera:

import pyzed.sl as sl
import os
import time
from datetime import datetime
import cv2
import numpy as np


def Get_left_eye_image():
    # Create a ZED camera object
    zed = sl.Camera()

    # Set configuration parameters
    init_params = sl.InitParameters()
    init_params.camera_resolution = sl.RESOLUTION.HD1080
    init_params.camera_fps = 30

    # Open the camera
    err = zed.open(init_params)
    if err != sl.ERROR_CODE.SUCCESS:
        exit(-1)
    
    image = sl.Mat()
    runtime_parameters = sl.RuntimeParameters()
    if zed.grab(runtime_parameters) == sl.ERROR_CODE.SUCCESS:
        zed.retrieve_image(image, sl.VIEW.LEFT)
    
    cv2.imshow("Hello", image.get_data())
    cv2.waitKey(0) 
    cv2.destroyAllWindows()
    return zed

def Get_video(zed):

    # Enable recording with the filename specified in argument
    output_path = "test_{:%Y_%m_%d_%H_%M_%S}.svo".format(datetime.now())
    recordingParameters = sl.RecordingParameters()
    recordingParameters.compression_mode = sl.SVO_COMPRESSION_MODE.H264
    recordingParameters.video_filename = output_path
    err = zed.enable_recording(recordingParameters)

    time_to_record = 10
    start = time.time()
    while True:
        # Each new frame is added to the SVO file
        if zed.grab() == sl.ERROR_CODE.SUCCESS:
            recording_time = time.time() - start
            print(f"Time recorded: {recording_time}", end="\r")
            if recording_time >= time_to_record - 0.0001:
                zed.disable_recording()
                break

if __name__ == "__main__":

    zed = Get_left_eye_image()
    Get_video(zed)

I have a linux 22.04 computer to which I recently installed the ZED SDK 4.1.0 and the code worked as expected. But now I have two problems:

Upon running the code from visual studio code, but this time attached to a container based on the “stereolabs/zed:4.1-devel-cuda12.1-ubuntu22.04” image, executed with:

docker run --gpus all --detach -it \
            --name PositionalTracking \
            --privileged\
            -e DISPLAY=$DISPLAY \
            -v $XSOCK:$XSOCK -v $XAUTH:$XAUTH -e XAUTHORITY=$XAUTH -v /home/wildsense2/Documents/ZED:/home/positional_tracking/videos -v ${PWD}:/home/positional_tracking\
            stereolabs/zed:4.1-devel-cuda12.1-ubuntu22.04

I get the next output:

[2024-04-19 20:30:41 UTC][ZED][INFO] Logging level INFO
[2024-04-19 20:30:41 UTC][ZED][INFO] [Init]  Depth mode: PERFORMANCE
[2024-04-19 20:30:42 UTC][ZED][INFO] [Init]  Camera successfully opened.
[2024-04-19 20:30:42 UTC][ZED][INFO] [Init]  Camera FW version: 1523
[2024-04-19 20:30:42 UTC][ZED][INFO] [Init]  Video mode: HD1080@30
[2024-04-19 20:30:42 UTC][ZED][INFO] [Init]  Serial Number: S/N 30230648
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/usr/local/lib/python3.10/dist-packages/cv2/qt/plugins" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb.

I tried to run the same code in a Jetson Orin Nano that has just been formatted with the jetpack 5.1.2 package. But upon getting to the line “err = zed.enable_recording(recordingParameters)” the OS completely freezes for about 1 minute before rebooting. This happened in several occasions and with different codes (that where previously functioning with SDK 4.0.8).
Previous to this I installed the corresponding ZED SDK 4.1 and tested the ZED_Depth_viewer and ZED_Explorer and everything went fine.

Thanks in advance!
PD: Feeling very enthusiastic with the new update!!!

NVG97 · April 22, 2024, 5:27pm

Update:
While trying to solve problem number one I replicated the process in another computer. To do this I followed the next steps:

Install ZED SDK 4.1 locally on top of previous version 4.0.8.
Verify local functionality of the test code. It worked.
Pull docker image:

docker pull stereolabs/zed:4.1-devel-cuda12.1-ubuntu22.04

Execute docker container with:

docker run --gpus all --detach -it \
            --name PositionalTracking \
            --privileged\
            -e DISPLAY=$DISPLAY \
            -v $XSOCK:$XSOCK -v $XAUTH:$XAUTH -e XAUTHORITY=$XAUTH -v /home/wildsense2/Documents/ZED:/home/positional_tracking/videos -v ${PWD}:/home/positional_tracking\
            stereolabs/zed:4.1-devel-cuda12.1-ubuntu22.04

Run test code, which gets the next error on opencv import:

Exception has occurred: ImportError
libgthread-2.0.so.0: cannot open shared object file: No such file or directory
  File "/home/positional_tracking/capture_frame_and_video.py", line 5, in <module>
    import cv2
ImportError: libgthread-2.0.so.0: cannot open shared object file: No such file or directory

Install missing dependency with the next commands:

sudo apt-get update
sudo apt-get install libglib2.0-0

Run test code again, which gets the next error with cv2.imshow():

[2024-04-22 17:23:20 UTC][ZED][INFO] Logging level INFO
[2024-04-22 17:23:21 UTC][ZED][INFO] [Init]  Depth mode: PERFORMANCE
[2024-04-22 17:23:22 UTC][ZED][INFO] [Init]  Camera successfully opened.
[2024-04-22 17:23:22 UTC][ZED][INFO] [Init]  Camera FW version: 1523
[2024-04-22 17:23:22 UTC][ZED][INFO] [Init]  Video mode: HD1080@30
[2024-04-22 17:23:22 UTC][ZED][INFO] [Init]  Serial Number: S/N 36848014
[2024-04-22 17:23:22 UTC][ZED][INFO] [Init]  No calibration file found for SN 36848014. Downloading... 
[2024-04-22 17:23:23 UTC][ZED][INFO] [Init]  Calibration file downloaded.
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/usr/local/lib/python3.10/dist-packages/cv2/qt/plugins" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb.

NVG97 · April 22, 2024, 7:41pm

Update:
Problem 1 is solved by installing missing packages:

sudo apt-get update
sudo apt-get install libsm6 libice6 libglib2.0-0

–
Problem two remains unsolved.

mattrouss · April 23, 2024, 10:13am

Hi @NVG97,

For the first issue, glad to hear that you could solve it. The issue was indeed the lack of graphical dependencies in the devel image. This is actually expected behavior for the devel image which is supposed to only run code headless in development, for graphical support we have other images, such as stereolabs/zed:4.1-gl-devel-cuda11.4-ubuntu20.04

However, we haven’t currently deployed the images for Ubuntu22.04. Thank you for reporting this, we will add this to our deployment CI to be consistent on releases.

For the second issue, I believe there are multiple factors at work here:

You are recording on Jetson Orin Nano, which does not have any hardware encoders and is only able to record in LOSSLESS mode.
With the new ZED SDK 4.1, we have upgraded our SVOs to include high-frequency data, but have noticed issues with recording performance specifically in LOSSLESS mode.

What I can suggest is to run the following command before running your recording script:

export ZED_SDK_SVO_VERSION=1

This will allow recording SVOs in version 1, which is the same format as ZED SDK 4.0.8 (and is still compatible with ZED SDK 4.1).

On our side, we are investigating the issue with SVO gen2 and lossless compression.

NVG97 · April 23, 2024, 2:13pm

Hi @mattrouss,

I tried changing the enviroment variable as seen in the next image

But the issue persists. Upon executing the .enable_recording() method, as shown in the next image:

The jetson completely freezes to then proceed to restart after 1 min app.
Am I setting it wrong?

NVG97 · April 23, 2024, 3:43pm

Update: I verified that the environment variable is set to 1 and implemented some writting to the syslog to try to see what happened in the background. The modified code is the following:

import pyzed.sl as sl
import os
import cv2
import logging
import syslog

# Configure the logger
logging.basicConfig(filename='zed.log', level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')
console = logging.StreamHandler()
console.setLevel(logging.INFO)  # Set the level for console logging
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
console.setFormatter(formatter)
logging.getLogger('').addHandler(console)

def Get_left_eye_image():
    try:
        # Create a ZED camera object
        zed = sl.Camera()

        # Set configuration parameters
        init_params = sl.InitParameters()
        init_params.camera_resolution = sl.RESOLUTION.HD1080
        init_params.camera_fps = 30

        # Open the camera
        err = zed.open(init_params)
        if err != sl.ERROR_CODE.SUCCESS:
            logging.error("Failed to open ZED camera.")
            exit(-1)
        
        image = sl.Mat()
        runtime_parameters = sl.RuntimeParameters()
        syslog.syslog(syslog.LOG_INFO, "ZED_DEBUGGER: zed.grab()")
        if zed.grab(runtime_parameters) == sl.ERROR_CODE.SUCCESS:
            zed.retrieve_image(image, sl.VIEW.LEFT)
        
        syslog.syslog(syslog.LOG_INFO, "ZED_DEBUGGER: cv2.imshow()")
        cv2.imshow("Hello", image.get_data())
        cv2.waitKey(0) 
        cv2.destroyAllWindows()
        return zed
    except Exception as e:
        logging.exception("An error occurred in Get_left_eye_image(): %s", e)

def Get_video(zed):
    try:
        # Enable recording with the filename specified in argument
        output_path = "hello.svo"
        recordingParameters = sl.RecordingParameters()
        recordingParameters.compression_mode = sl.SVO_COMPRESSION_MODE.H265
        recordingParameters.video_filename = output_path

        
        syslog.syslog(syslog.LOG_INFO, "ZED_DEBUGGER: Start to record ...")
        err = zed.enable_recording(recordingParameters)
        

        i = 0
        while not i < 60:
            # Each new frame is added to the SVO file
            zed.grab()
            i += 1

        # Disable recording
        zed.disable_recording()
    except Exception as e:
        logging.exception("An error occurred in Get_video(): %s", e)

if __name__ == "__main__":
    syslog.openlog()
    syslog.syslog(syslog.LOG_INFO, "ZED_DEBUGGER: Program started ...")
    syslog.syslog(syslog.LOG_INFO, "ZED_DEBUGGER: Program started 2...")
    env_var = "ZED_SDK_SVO_VERSION"
    print("{} = {}".format(env_var, os.getenv(env_var)))

    zed = Get_left_eye_image()
    if zed:
        Get_video(zed)
    
    syslog.closelog()

The portion of the syslog that corresponds to the execution of the code doesn’t show what happens after the .enable_recording() method. But just in case I’ll leave it here:

Apr 23 11:28:38 wildsenseorin-desktop /zed_debugger.py: ZED_DEBUGGER: Program started ...
Apr 23 11:28:38 wildsenseorin-desktop /zed_debugger.py: ZED_DEBUGGER: Program started 2...
Apr 23 11:28:40 wildsenseorin-desktop kernel: [  723.848604] tegra-i2c 3190000.i2c: I2C transfer timed out
Apr 23 11:28:40 wildsenseorin-desktop kernel: [  723.960593] tegra-i2c 3190000.i2c: I2C transfer timed out
Apr 23 11:28:40 wildsenseorin-desktop kernel: [  724.072613] tegra-i2c 31b0000.i2c: I2C transfer timed out
Apr 23 11:28:40 wildsenseorin-desktop kernel: [  724.184590] tegra-i2c 31b0000.i2c: I2C transfer timed out
Apr 23 11:28:40 wildsenseorin-desktop kernel: [  724.296595] tegra-i2c 31c0000.i2c: I2C transfer timed out
Apr 23 11:28:40 wildsenseorin-desktop kernel: [  724.408601] tegra-i2c 31c0000.i2c: I2C transfer timed out
Apr 23 11:28:40 wildsenseorin-desktop kernel: [  724.520597] tegra-i2c 31e0000.i2c: I2C transfer timed out
Apr 23 11:28:40 wildsenseorin-desktop kernel: [  724.632588] tegra-i2c 31e0000.i2c: I2C transfer timed out
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4651]: hiddev1: Process '/usr/lib/snapd/snap-device-helper snap_cups_cupsd' failed with exit code 1.
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4651]: hiddev1: Process '/usr/lib/snapd/snap-device-helper snap_cups_ippeveprinter' failed with exit code 1.
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4651]: 1-2.3.2:1.0: Process '/usr/lib/snapd/snap-device-helper snap_cups_cupsd' failed with exit code 1.
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4651]: 1-2.3.2:1.0: Process '/usr/lib/snapd/snap-device-helper snap_cups_ippeveprinter' failed with exit code 1.
Apr 23 11:28:42 wildsenseorin-desktop kernel: [  726.304852] usb 1-2.3.2: reset full-speed USB device number 8 using tegra-xusb
Apr 23 11:28:42 wildsenseorin-desktop kernel: [  726.419184] hid-generic 0003:2B03:F881.0005: hiddev97,hidraw3: USB HID v1.11 Device [STEREOLABS ZED-2i HID INTERFACE] on usb-3610000.xhci-2.3.2/input0
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4651]: 1-2.3.2: Process '/usr/lib/snapd/snap-device-helper snap_cups_cupsd' failed with exit code 1.
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4651]: 1-2.3.2: Process '/usr/lib/snapd/snap-device-helper snap_cups_ippeveprinter' failed with exit code 1.
Apr 23 11:28:42 wildsenseorin-desktop upowerd[2833]: treating change event as add on /sys/devices/platform/3610000.xhci/usb1/1-2/1-2.3/1-2.3.2
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4651]: 1-2.3.2: Process '/usr/lib/snapd/snap-device-helper snap_cups_cupsd' failed with exit code 1.
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4651]: 1-2.3.2: Process '/usr/lib/snapd/snap-device-helper snap_cups_ippeveprinter' failed with exit code 1.
Apr 23 11:28:42 wildsenseorin-desktop upowerd[2833]: treating change event as add on /sys/devices/platform/3610000.xhci/usb1/1-2/1-2.3/1-2.3.2
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4652]: hiddev1: Process '/usr/lib/snapd/snap-device-helper snap_cups_cupsd' failed with exit code 1.
Apr 23 11:28:42 wildsenseorin-desktop fido_id[4665]: 1-2.3.2:1.0: Failed to open report descriptor at '/sys/devices/platform/3610000.xhci/usb1/1-2/1-2.3/1-2.3.2/1-2.3.2:1.0/report_descriptor': No such file or directory
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4652]: hiddev1: Process '/usr/lib/snapd/snap-device-helper snap_cups_ippeveprinter' failed with exit code 1.
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4651]: 1-2.3.2:1.0: Process '/usr/lib/snapd/snap-device-helper snap_cups_cupsd' failed with exit code 1.
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4651]: 1-2.3.2:1.0: Process '/usr/lib/snapd/snap-device-helper snap_cups_ippeveprinter' failed with exit code 1.
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4651]: hiddev1: Process '/usr/lib/snapd/snap-device-helper snap_cups_cupsd' failed with exit code 1.
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4651]: hiddev1: Process '/usr/lib/snapd/snap-device-helper snap_cups_ippeveprinter' failed with exit code 1.
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4651]: 1-2.3.2:1.0: Process '/usr/lib/snapd/snap-device-helper snap_cups_cupsd' failed with exit code 1.
Apr 23 11:28:42 wildsenseorin-desktop systemd-udevd[4651]: 1-2.3.2:1.0: Process '/usr/lib/snapd/snap-device-helper snap_cups_ippeveprinter' failed with exit code 1.
Apr 23 11:28:44 wildsenseorin-desktop /zed_debugger.py: ZED_DEBUGGER: zed.grab()
Apr 23 11:28:44 wildsenseorin-desktop /zed_debugger.py: ZED_DEBUGGER: cv2.imshow()
Apr 23 11:28:46 wildsenseorin-desktop /zed_debugger.py: ZED_DEBUGGER: Start to record ...
Apr 23 11:25:27 wildsenseorin-desktop systemd-modules-load[278]: Inserted module 'nvmap'

I’ll try in another jetson Orin Nano and to go back to SDK 4.0.8 to see if the problem persists.

mattrouss · April 25, 2024, 12:33pm

Hi @NVG97,

Sorry for the late reply, and it appears I was not clear in my previous message.

The only compression mode available on Jetson Orin Nano is the SVO_COMPRESSION_MODE::LOSSLESS. Can you please set this mode in your script and try again?

NVG97 · April 25, 2024, 2:17pm

Thank you! That solved it.

What Jetsons are able to run H265 compression mode?

mattrouss · April 25, 2024, 5:11pm

Glad to hear this was the issue.

Jetson Orin Nano is the only one with this constraint as it does not have any hardware encoders to perform H264/H265 encoding on its GPU.

NVG97 · April 29, 2024, 3:28pm

Hi again @mattrouss,

I would like to know if there are any plans of being able to achieve LOSSY compression modes using only software, hence being able to use H265/H264 compression modes on the Jetson Orin Nano. The company already purchased several Orin Nano anticipating to use them as main hardware in the project I’m workin on and part of that implied to use .svo files small in size.

Additionally, Where in the documentation is it said that LOSSLESS compression is the only available mode for Orin Nano?

Thank you again for your time.

mattrouss · April 30, 2024, 6:09pm

Hi @NVG97,

I’ve asked our technical team and we are indeed planning software encoding for the next release of the ZED SDK, specifically for your use case using the Orin Nano which lacks hardware encoding.

I would like to mention that this type of encoding will come with a non-negligible CPU overhead.

I agree with you, the documentation is not clear that an NVENC hardware encoder is required, which the Orin Nano lacks. I’ve logged the issue to fix the crash that you’ve reported and provide more user friendly logs on why the recording does not work, and also improve the documentation related to this.