Need Help with Streaming Error: [Streaming] Metadata timeout. the size is equal to 196 instead of 21960. Skipping

Software
1

Hi, I am trying to set up 4 Zed X cameras to stream over IP. I have a couple issues I am trying to debug.

The first is the title of this post. When I try to connect from a remote machine, I get this error. I have looked around the forum, but did not see a lot about this and even less of a solution. However, I think it has something to do with the communication with the two computers because I can subscribe to the stream locally on the same computer sending it and see the feed just fine. I have verified that both computers are able to ping each other.

The second is a little more random and I am not sure if it warrants its own topic. I have used the logic from the ‘sender’ and ‘multi-camera’ depth examples to create a script that launches all 4 zeds and streams them over IP. The script seems to be working and I can subscribe to each camera(locally), but I occasionally and seemingly randomly get a segmentation fault and the cameras are shut down. It does seem to be closing somewhat gracefully as I am able to reconnect, but I am unsure where this is coming from. Any suggestions on how to debug?

Thanks!
Benjamin

2

Hi @benijohn
Welcome to the Stereolabs community.

Are you using a VPN? If not, please check the MTU configuration of the network adapters and change it to the maximum allowed size.

Can you provide more information about this problem?
What ZED SDK version are you using?
What type of error do you get when the software crashes?
Can you send the full console log?

3

@Myzhar thank you for your reply. Indeed I am using VPN and the MTU was the issue. I can now connect remotely to the streams.

I am however still suffering from the seemingly random segmentation faults.
The system specs are:
Sender: Jetson Orign AGX 64, SDK 4.2.2, Jetpack 5.1.1, CUDA 11.4
Reciever: Ubuntu 20.04, SDK 4.2.3, CUDA 12.4

Unfortunately, when I get the segmentation fault, I am not getting any output. It has generally let me immediately run the streaming script again to reconnect, however this morning. When trying to reconnect I got the following error:


  (Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 219)
(Argus) Error Timeout: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)
[ZED-X][Warning] Failed to connect to zed_x_daemon.
No ZED Detected, exit program

But when it starts normally I get the following output:

ID : 0 ,model : ZED X , S/N : 48935869 , state : AVAILABLE
ID : 1 ,model : ZED X , S/N : 44036609 , state : AVAILABLE
ID : 2 ,model : ZED X , S/N : 49440015 , state : AVAILABLE
ID : 3 ,model : ZED X , S/N : 41385721 , state : AVAILABLE
4 ZED Detected
[2025-01-22 12:48:55 UTC][ZED][INFO] Logging level INFO
[2025-01-22 12:48:55 UTC][ZED][INFO] Logging level INFO
[2025-01-22 12:48:55 UTC][ZED][INFO] Logging level INFO
[2025-01-22 12:48:55 UTC][ZED][INFO] Logging level INFO
[ZED-X][Warning] Failed to connect to zed_x_daemon.
ZED X, ID: 0, SN: 48935869 Opened
ZED X, ID: 1, SN: 44036609 Opened
ZED X, ID: 2, SN: 49440015 Opened
ZED X, ID: 3, SN: 41385721 Opened
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
NVMEDIA: H265 : Profile : 1 
[Sample] Camera 48935869 streaming on port 30000 
NVMEDIA: Need to set EMC bandwidth : 2872000 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
NVMEDIA: H265 : Profile : 1 
[Sample] Camera 44036609 streaming on port 30002 
NVMEDIA: Need to set EMC bandwidth : 2872000 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
NVMEDIA: H265 : Profile : 1 
[Sample] Camera 49440015 streaming on port 30004 
NVMEDIA: Need to set EMC bandwidth : 2872000 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
NVMEDIA: H265 : Profile : 1 
[Sample] Camera 41385721 streaming on port 40000 
NVMEDIA: Need to set EMC bandwidth : 2872000 
NVMEDIA_ENC: bBlitMode is set to TRUE 
NVMEDIA_ENC: bBlitMode is set to TRUE 
NVMEDIA_ENC: bBlitMode is set to TRUE 
NVMEDIA_ENC: bBlitMode is set to TRUE 
[Streaming] Streaming is now running.... 
[Streaming] Streaming is now running.... 
[Streaming] Streaming is now running.... 
[Streaming] Streaming is now running....

which is taken directly from the provided examples. I am attaching the script I used to launch the cameras. It is a combination of the streaming example and the multi-camera depth example. Criticism is welcome.

main.cpp (9.4 KB)

Thank you!
Benjamin

4

I guess one more thing. It does seem that the segmentation fault appears when connecting/disconnection (or attempting to) from the remote machine. The remote machine is using the zed-ros-wrapper to connect to the streams and 50% of the time it launches everything fine. The other 50% of the time it launches with a warning that Positional tracking is not started.

[WARNING] Positional Tracking GEN2 with IMU fusion enabled requires high frequency sensors data (available with Streaming version 2, with ZED SDK >= 4.1). IMU Fusion will be disabled, performance will be degraded.

I am not sure if this is related to the issue with the segmentation fault or not.

Thanks!
Benjamin

5

This normally happens when the ZED X Driver is not correctly initialized.
You can force a reset of the connection by using the command
sudo service zed_x_daemon restart
after a few seconds (i.e. 5 seconds) the connection should normally be recovered.

Everything here is good. Unfortunately, we cannot prevent the NVMEDIA library to write on the standard output.

Your code seems good to me, I only recommend a better error handling process here:

        if (zed.grab() != ERROR_CODE::SUCCESS) {
            sleep_ms(1);
        }

I recommend you use GEN_1 as positional tracking mode when using the stream. GEN_2 is not stable now (we will release many fixes soon) and it also does not work correctly if the IMU information at not received at the maximum allowed rate (i.e. 400 Hz in live mode).

It’s possible that GEN_2 is the cause of these crashes.

6

@Myzhar I am happy to try using GEN 1. Ultimately our tracking will come from GNSS anyway, so we will use static transforms to locate the cameras.

How do I force the SDK to use GEN 1?

Thanks!
Benjamin

7

This is the parameter to change in ROS: zed-ros-wrapper/zed_wrapper/params/common.yaml at master · stereolabs/zed-ros-wrapper · GitHub
It reflects the sl::PositionalTrackingParameters:: mode parameter