ZED SDK Stream depth mode

ergouu · July 15, 2024, 6:09am

Hi, I am using 3 Zedx with Jetson Orin AGX, and trying to stream via local network by the Sample.
Hardware:
Zed X *3 with Jetson Orin AGX, PC with RTX 3090.
Platform:
Sender: Jetson Orin AGX JP5.1.2 with a container runing ZED SDK 4.1
Reciever: PC Ubuntu 22.04 with a container runing ZED SDK 4.1
Problems:
I set the sender Depth Mode as NEURAL_PLUS , but the reciever get the init parameter from stream as PERFORMANCE. The point cloud is also PERFORMANCE mode.

sender.py as follows:

import pyzed.sl as sl

init=sl.InitParameters()
init.camera_resolution=sl.RESOLUTION.HD1080
init.camera_fps=30
init.coordinate_units=sl.UNIT.METER
init.depth_maximum_distance=20
# init.depth_mode=sl.DEPTH_MODE.ULTRA
init.depth_mode=sl.DEPTH_MODE.NEURAL_PLUS

zed_list=[]
name_list=[]

cameras=sl.Camera.get_device_list()
for idx,cam in enumerate(cameras):
    init.set_from_serial_number(cam.serial_number)
    zed_list.append(sl.Camera())
    name_list.append(cam.serial_number)
    status=zed_list[idx].open(init)
    zed_list[idx].set_camera_settings(sl.VIDEO_SETTINGS.EXPOSURE,-1)
    zed_list[idx].set_camera_settings(sl.VIDEO_SETTINGS.WHITEBALANCE_AUTO,1)
    if status != sl.ERROR_CODE.SUCCESS:
        print(repr(status))
        zed_list[idx].close()

stream=sl.StreamingParameters()
stream.codec=sl.STREAMING_CODEC.H264
stream.bitrate=8000
stream.port=30000

for zed in zed_list:
    err=zed.enable_streaming(stream)
    stream.port+=2
    
while True:
    for zed in zed_list:
        zed.grab()
pass

here is the senders logs:

[2024-07-15 05:19:22 UTC][ZED][INFO] Logging level INFO
[2024-07-15 05:19:23 UTC][ZED][INFO] [Init]  Depth mode: NEURAL PLUS
[2024-07-15 05:19:26 UTC][ZED][INFO] [Init]  Camera FW version: 2001
[2024-07-15 05:19:26 UTC][ZED][INFO] [Init]  Video mode: HD1080@30
[2024-07-15 05:19:26 UTC][ZED][INFO] [Init]  Serial Number: S/N 47253491
[2024-07-15 05:19:29 UTC][ZED][INFO] Logging level INFO
[2024-07-15 05:19:29 UTC][ZED][INFO] [Init]  Depth mode: NEURAL PLUS
[2024-07-15 05:19:32 UTC][ZED][INFO] [Init]  Camera FW version: 2001
[2024-07-15 05:19:32 UTC][ZED][INFO] [Init]  Video mode: HD1080@30
[2024-07-15 05:19:32 UTC][ZED][INFO] [Init]  Serial Number: S/N 45705924
[2024-07-15 05:19:33 UTC][ZED][INFO] Logging level INFO
[2024-07-15 05:19:33 UTC][ZED][INFO] [Init]  Depth mode: NEURAL PLUS
[2024-07-15 05:19:37 UTC][ZED][INFO] [Init]  Camera FW version: 2001
[2024-07-15 05:19:37 UTC][ZED][INFO] [Init]  Video mode: HD1080@30
[2024-07-15 05:19:37 UTC][ZED][INFO] [Init]  Serial Number: S/N 42570003
[2024-07-15 05:19:38 UTC][ZED][INFO] [Init]  Notice: The streaming is using STREAM version 2, enabled by default starting from SDK version 4.1. To revert to the original stream version, set the environment variable "ZED_SDK_STREAM_VERSION" to 1
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
===== NvVideo: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 77, Level = 50 
[2024-07-15 05:19:38 UTC][ZED][INFO] [Init]  Notice: The streaming is using STREAM version 2, enabled by default starting from SDK version 4.1. To revert to the original stream version, set the environment variable "ZED_SDK_STREAM_VERSION" to 1
NVMEDIA: Need to set EMC bandwidth : 2872000 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
===== NvVideo: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 77, Level = 50 
[2024-07-15 05:19:38 UTC][ZED][INFO] [Init]  Notice: The streaming is using STREAM version 2, enabled by default starting from SDK version 4.1. To revert to the original stream version, set the environment variable "ZED_SDK_STREAM_VERSION" to 1
NVMEDIA: Need to set EMC bandwidth : 2872000 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
===== NvVideo: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 77, Level = 50 
NVMEDIA: Need to set EMC bandwidth : 2872000 
NvVideo: bBlitMode is set to TRUE 
NvVideo: bBlitMode is set to TRUE 
NvVideo: bBlitMode is set to TRUE 
[Streaming] Streaming is now running.... 
[Streaming] Streaming is now running.... 
[Streaming] Streaming is now running.... 
[ZED][Streaming] Adding Receiver with IP: 192.168.10.134
[ZED][Streaming] Adding Receiver with IP: 192.168.10.242
[ZED][Streaming] Removing Receiver with IP: 192.168.10.242
[ZED][Streaming] Adding Receiver with IP: 192.168.10.242
[ZED][Streaming] Adding Receiver with IP: 192.168.10.242
[ZED][Streaming] Adding Receiver with IP: 192.168.10.242
[ZED][Streaming] Removing Receiver with IP: 192.168.10.242
[ZED][Streaming] Removing Receiver with IP: 192.168.10.242
[ZED][Streaming] Removing Receiver with IP: 192.168.10.242
[ZED][Streaming] Adding Receiver with IP: 192.168.10.134
[ZED][Streaming] Adding Receiver with IP: 192.168.10.134
[ZED][Streaming] Adding Receiver with IP: 192.168.10.134

the reciever.py:

class Cameras:
    def __init__(self,ip_list:list,port_list:list) -> None:
        # ip_list: camera ip list
        # port_list: camera port list
        self._zed_list=[]
        for ip,port in zip(ip_list,port_list):
            init=sl.InitParameters()
            init.set_from_stream(ip,port)
            zed=sl.Camera()
            err=zed.open(init)
            if err != sl.ERROR_CODE.SUCCESS:
                exit(1)
            self._zed_list.append(zed)
    def get_img(self,get_pcd=False,get_depth=False):
        img_list=[]
        pcd_list=[]
        dep_list=[]
        serial_number_list=[]
        img=sl.Mat()
        pcd=sl.Mat()
        depth=sl.Mat()
        for zed in self._zed_list:
            if zed.grab() == sl.ERROR_CODE.SUCCESS:
                zed.retrieve_image(img,sl.VIEW.LEFT)
                img_ocv=img.get_data()
                img_list.append(img_ocv.copy())
                serial_number_list.append(zed.get_camera_information().serial_number)
                if get_pcd:
                    zed.retrieve_measure(pcd,sl.MEASURE.XYZRGBA)
                    pcd_ocv=pcd.get_data()
                    pcd_list.append(pcd_ocv.copy())
                if get_depth:
                    zed.retrieve_measure(depth,sl.MEASURE.DEPTH)
                    dep_ocv=depth.get_data()
                    dep_list.append()(dep_ocv.copy())
        data={
            'serial_number':serial_number_list,
            'img':img_list,
            'pcd':pcd_list,
            'dep':dep_list
        }
        return data

if __name__=='__main__':
    cams=Cameras(['192.168.10.242','192.168.10.242','192.168.10.242'],[30000,30002,30004])
    data=cams.get_img(get_pcd=True)
    pass

the reciever logs:

[2024-07-15 05:42:12 UTC][ZED][INFO] Logging level INFO
[Streaming] Warning : receiving port 30000 is not available (already used)... switching to port 30002. Retrying...
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] No backward compatibility required.
[Streaming] Warning: Corrupted frame chunk received (recv: 196 / expect 16100)  from ip : 192.168.10.242 at ts 1721022132929(ms)
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[2024-07-15 05:42:14 UTC][ZED][INFO] [Init]  Depth mode: PERFORMANCE
[2024-07-15 05:42:14 UTC][ZED][INFO] [Init]  Serial Number: S/N 47253491
[2024-07-15 05:42:16 UTC][ZED][INFO] Logging level INFO
[Streaming] Warning : receiving port 30002 is not available (already used)... switching to port 30004. Retrying...
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] No backward compatibility required.
[Streaming] Warning: Corrupted frame chunk received (recv: 196 / expect 16100)  from ip : 192.168.10.242 at ts 1721022136070(ms)
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[2024-07-15 05:42:17 UTC][ZED][INFO] [Init]  Depth mode: PERFORMANCE
[2024-07-15 05:42:17 UTC][ZED][INFO] [Init]  Serial Number: S/N 45705924
[2024-07-15 05:42:18 UTC][ZED][INFO] Logging level INFO
[Streaming] Warning : receiving port 30004 is not available (already used)... switching to port 30006. Retrying...
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] No backward compatibility required.
[Streaming] Warning: Corrupted frame chunk received (recv: 196 / expect 16100)  from ip : 192.168.10.242 at ts 1721022138437(ms)
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping.
[2024-07-15 05:42:20 UTC][ZED][INFO] [Init]  Depth mode: PERFORMANCE
[2024-07-15 05:42:20 UTC][ZED][INFO] [Init]  Serial Number: S/N 42570003
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.
[Streaming] Metadata timeout. the size is equal to 16100 instead of 21960. Skipping.

Myzhar · July 15, 2024, 8:14am

Hi @ergouu

It is not important how the sender is configured, you could also disable the depth processing on the sender side. Only resolution and frame rate are fixed by the sender.
The receiver indeed is as if it’s working with a camera directly connected to it.

ergouu · July 15, 2024, 9:56am

Thx for reply @Myzhar.
So that means the camera sender just send the left and right views, and the depth actually is computed on the reciever machine. And I also can set the reciever depth mode by setting init.depth_mode=sl.DEPTH_MODE.NEURAL_PLUS , after init.set_from_stream(ip,port)?

Myzhar · July 15, 2024, 10:18am

This is exact.

Yes