Hi everyone,
We tried to get video stream with Zed X mini using Live Link. But, there’s about 1 minute delay in streaming. How can we increase the performance and reduce the delay? The delay must be under 1 second in our project.
(We are using NVIDIA Jetson Orin 64GB developer kit.)
Cheers.
Hi, welcome to the community!
One minute seems crazy. You should have less than a second. something like 200ms is the most common situation. Can you explain in details how you are trying to do this?
Hi @alassagne,
Thank you for your response. Apologies for the confusion in my previous message. Our video streaming setup did not involve Live Link with a 1-minute delay as initially stated; instead, it relied on specific pieces of code. To enhance performance, we have now transitioned to using Live Link.
In our project, we’re dealing with remote video streaming from the Zed X mini camera, rather than local streaming. Following the guidelines outlined in the documentation (Live Link for UE5 - Stereolabs), we attempted to implement the JSON file with the provided code snippet:
{
“InitParameters”:
{
“input”: “STREAM”,
“input_path”: 192.168.1.170,
“resolution”: “HD1080”,
“fps”: 60,
“depth_mode”: “ULTRA”
},
“PositionalTrackingParameters”:
{
“enable_pose_smoothing” : true,
“enable_area_memory”: false
},
“BodyTrackingParameters”:
{
“enable_module”: false,
“detection_model”: “HUMAN_BODY_ACCURATE”,
“body_format”: “BODY_38”,
“body_selection”: “FULL”,
“confidence”: 40,
“max_range”: -1,
“minimum_keypoints_threshold” : -1,
“skeleton_smoothing”: 0.2
}
}
However, upon execution, we encountered the following error message: “FAILURE: Standard code for unsuccessful behavior.”
We think that the issue may lie with the JSON file. Given the scarcity of documentation on this particular problem, we seek assistance in understanding how to properly configure the initialization parameters within the JSON file to successfully access the camera.
Any guidance or insights you could provide would be greatly appreciated. Thank you for your attention to this matter.
Hi @NeginHeiran,
You’re right, the doc is lacking on this matter, we’ll work on this.
input_path should be between quotes, else it won’t be able to parse the JSON correctly.Hi @JPlou,
So, is that possible to use Live Link remotely for now?
Please kindly let me know how I can use Zed X remotely if Live Link has issues with it currently.
Best.
@NeginHeiran
I do reproduce the issue here, I think.
I will look into this as soon as possible and update the sample if needed, there is no reason it should not work.
Hi @JPlou,
Thank you for the follow-up. Here’s the log note from sender:
Created: Sat Feb 17 2024
###########################################
############ DMESG [ZEDX] LOG #############
###########################################
[ 20.213700] zedx 30-0018: Driver Version : v0.5.4
[ 20.213709] zedx 30-0018: Probing v4l2 sensor.
[ 20.213958] zedx 30-0018: tegracam sensor driver:video0_v2.0.6
[ 27.289193] zedx 30-0018: zedx_links_check: 1 link(s) detected
[ 30.212290] zedx 30-0018: Single ZED-X camera detect success
[ 30.440766] zedx 30-0018: zedx_set_frame_rate: Unsupported value
[ 30.769063] tegra-camrtc-capture-vi tegra-capture-vi: subdev zedx 30-0018 bound
[ 30.770565] zedx 30-0018: Detected ZED-X sensor
[ 30.770847] zedx 30-0010: Driver Version : v0.5.4
[ 30.770852] zedx 30-0010: Probing v4l2 sensor.
[ 30.771109] zedx 30-0010: tegracam sensor driver:video1_v2.0.6
[ 30.785123] zedx 30-0018: zedx_open: Accessing the camera
[ 30.877412] zedx 30-0010: zedx_links_check: 1 link(s) detected
[ 31.102952] zedx 30-0010: zedx_set_frame_rate: Unsupported value
[ 31.432660] tegra-camrtc-capture-vi tegra-capture-vi: subdev zedx 30-0010 bound
[ 31.433791] zedx 30-0010: Detected ZED-X sensor
[ 31.433951] zedx 31-0018: Driver Version : v0.5.4
[ 31.433953] zedx 31-0018: Probing v4l2 sensor.
[ 31.434117] zedx 31-0018: tegracam sensor driver:video2_v2.0.6
[ 31.437164] zedx 30-0010: zedx_open: Accessing the camera
[ 31.541393] zedx 31-0018: zedx_links_check: 1 link(s) detected
[ 31.542083] zedx 31-0018: zedx_write_reg:i2c write failed, 0x301a = 2058
[ 31.549050] zedx 31-0018: ZED-X detect error
[ 31.549186] zedx: probe of 31-0018 failed with error -1
[ 31.554668] zedx 31-0010: Driver Version : v0.5.4
[ 31.554672] zedx 31-0010: Probing v4l2 sensor.
[ 31.554971] zedx 31-0010: tegracam sensor driver:video3_v2.0.6
[ 31.661472] zedx 31-0010: zedx_links_check: 1 link(s) detected
[ 31.662067] zedx 31-0010: zedx_write_reg:i2c write failed, 0x301a = 2058
[ 31.669062] zedx 31-0010: ZED-X detect error
[ 31.669197] zedx: probe of 31-0010 failed with error -1
[ 31.981098] zedx 30-0010: zedx_open: Accessing the camera
[ 31.981174] zedx 30-0018: zedx_open: Accessing the camera
[95471.781559] zedx 30-0010: zedx_open: Accessing the camera
[95471.785278] zedx 30-0018: zedx_open: Accessing the camera
###########################################
############ DMESG [DTS] LOG #############
###########################################
[ 0.004259] DTS File Name: /home/louis/Documents/jetson/QA/35.3/jetson-gmsl-kernel/src/kernel/kernel-5.10/arch/arm64/boot/dts/…/…/…/…/…/…/hardware/nvidia/platform/t23x/concord/kernel-dts/tegra234-p3701-0000-p3737-0000.dts
[ 2.504698] tegra-pmc c360000.pmc: scratch reg offset dts data not present
###########################################
############ DTB list original ###########
###########################################
/boot/dtb/kernel_tegra234-p3701-0000-p3737-0000.dtb
###########################################
############ DTB list SLabs ###########
###########################################
/boot/stereolabs/tegra194-p2888-0001-p2822-0000.dtb
/boot/stereolabs/tegra194-p3668-0000-p3509-0000.dtb
/boot/stereolabs/tegra194-p3668-0001-p3509-0000.dtb
/boot/stereolabs/tegra234-p3701-0000-p3737-0000.dtb
/boot/stereolabs/tegra234-p3767-0000-p3509-a02.dtb
/boot/stereolabs/tegra234-p3767-0000-p3768-0000-a0.dtb
/boot/stereolabs/tegra234-p3767-0001-p3509-a02.dtb
/boot/stereolabs/tegra234-p3767-0003-p3768-0000-a0.dtb
/boot/stereolabs/utils:
extlinux.py
fio.py
pycache
###########################################
############ EXTlinux file ###########
###########################################
TIMEOUT 30
DEFAULT Stereolabs
MENU TITLE L4T boot options
LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
FDT /boot/dtb/kernel_tegra234-p3701-0000-p3737-0000.dtb
INITRD /boot/initrd
APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 console=ttyAMA0,115200 console=tty0 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0 nv-auto-config
LABEL Stereolabs
MENU LABEL Stereolabs kernel
LINUX /boot/Image
FDT /boot/stereolabs/tegra234-p3701-0000-p3737-0000.dtb
INITRD /boot/initrd
APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 console=ttyAMA0,115200 console=tty0 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0 nv-auto-config
####################################
############ Driver list ###########
####################################
/usr/lib/modules/5.10.104-tegra/kernel/drivers/stereolabs/max96712/max96712.ko==> exists
/usr/lib/modules/5.10.104-tegra/kernel/drivers/stereolabs/zedx/sl_zedx.ko ==> exists
################################################
############ ZEDX Daemon Status LS #############
################################################
● zed_x_daemon.service - ZED-X Daemon service
Loaded: loaded (/etc/systemd/system/zed_x_daemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2024-02-12 05:57:45 EST; 5 days ago
Main PID: 1186 (ZEDX_Daemon)
Tasks: 3 (limit: 74725)
Memory: 18.5M
CGroup: /system.slice/zed_x_daemon.service
└─1186 /usr/sbin/ZEDX_Daemon
Warning: journal has been rotated since unit was started, output may be incomplete.
################################################
############ I2C Trace [-l] #############
################################################
i2c-3 i2c 3190000.i2c I2C adapter
i2c-30 i2c i2c-2-mux (chan_id 0) I2C adapter
i2c-1 i2c c240000.i2c I2C adapter
i2c-8 i2c 31e0000.i2c I2C adapter
i2c-6 i2c 31c0000.i2c I2C adapter
i2c-33 i2c i2c-2-mux (chan_id 3) I2C adapter
i2c-4 i2c Tegra BPMP I2C adapter I2C adapter
i2c-31 i2c i2c-2-mux (chan_id 1) I2C adapter
i2c-2 i2c 3180000.i2c I2C adapter
i2c-0 i2c 3160000.i2c I2C adapter
i2c-9 i2c NVIDIA SOC i2c adapter 0 I2C adapter
i2c-7 i2c c250000.i2c I2C adapter
i2c-5 i2c 31b0000.i2c I2C adapter
i2c-32 i2c i2c-2-mux (chan_id 2) I2C adapter
################################################
############ I2C Trace /dev/i2C-30 #############
################################################
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: – – – – – – – – – – – – –
10: UU – – – – – – – UU 19 – – – – – –
20: – – – – – – – – – UU – – – – – –
30: – – – – – – – – – – – – – – – –
40: – – – – – – – – – 49 – – – – – –
50: – – – – 54 55 – – 58 59 – – – – – –
60: – – 62 – – – – – – 69 – – – – – –
70: UU – – – – – – –
################################################
############ I2C Trace /dev/i2C-31 #############
################################################
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: – – – – – – – – – – – – –
10: – – – – – – – – – – – – – – – –
20: – – – – – – – – – 29 – – – – – –
30: – – – – – – – – – – – – – – – –
40: – – – – – – – – – – – – – – – –
50: – – – – – – – – – – – – – – – –
60: – – – – – – – – – – – – – – – –
70: UU – – – – – – –
################################################
############ I2C Trace /dev/i2C-32 #############
################################################
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: – – – – – – – – – – – – –
10: – – – – – – – – – – – – – – – –
20: – – – – – – – – – – – – – – – –
30: – – – – – – – – – – – – – – – –
40: – – – – – – – – – – – – – – – –
50: – – – – – – – – – – – – – – – –
60: – – – – – – – – – – – – – – – –
70: UU – – – – – – –
################################################
############ I2C Trace /dev/i2C-33 #############
################################################
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: – – – – – – – – – – – – –
10: – – – – – – – – – – – – – – – –
20: – – – – – – – – – – – – – – – –
30: – – – – – – – – – – – – – – – –
40: – – – – – – – – – – – – – – – –
50: – – – – – – – – – – – – – – – –
60: – – – – – – – – – – – – – – – –
70: UU – – – – – – –
#########################
@NeginHeiran
Thanks for the logs!
It seems like the issue is before Live Link, in the sender on the jetson.
What resolution/fps are you trying to stream?
zedx_set_frame_rate: Unsupported value
This line indicates an unsupported setting in the sender. At the core, there may be a fallback value being used, creating a mismatch with the expected value in Live Link.
Dear @JPlou,
Thank you too!
We used 60fps, for your reference I attached the part of the code about resolution.
Can you confirm that your setup is:
Is that correct?
Then, some questions:
Sorry, that’s a bunch of questions, but I hope they’ll help pinpoint the issue because I don’t get the FAILURE message on my side.
Hi @JPlou,
The zed X mini camera is connected to the Jetson which is listening so that a windows or a PC connect to it.
On the other side, there’s a windows PC which want to connect to the Jetson through live link.
Thus, I think I can confirm what you said at first.
On both systems (windows and Jetson) SDK 4 has been installed.
Both devices have 60fps and 1080 resolution.
We are using Python streaming sample code on Jetson.
We have a failure error on the Windows side when pc wants to connect to the Jetson through Live Link. I attached the screenshot of the error and settings. The settings in Windows and Jetson are the same.
Cheers,
Negin
To be sure:
Also!
I missed your last answer in your other post, that could have helped you I think, sorry.
While the Live Link sender is currently not compatible with Jetson, we’ve developed a version that works independently of the Unreal source files and thus can be used directly on Jetson.
It’s currently on this branch on the repo, not documented yet, and not guaranteed bug-free yet, but it can be worth a try in your case I think.
Dear @JPlou,
We were using the old version of the repo that you mentioned. So, after applying the changes, the Jetson is running video streaming but the PC can’t connect to the Zed X. Here’s the result from the PC:
So, what’s the problem here? Do we need to do extra operations?
Also, please kindly let me know about the possibility of running video streaming using Live Link or is there any other method for video streaming without delay? Because our project deadline is near. Hence, our team needs to run the video streaming very soon.
Best Regards,
Negin
Dear @NeginHeiran,
The process is:
From what I see, that’s what you’ve done.
cmake .., make.the possibility of running video streaming using Live Link
You will not have access to the video feed using Live Link, only body tracking data. To get the video feed in UE5, please use our UE5 Plugin and stream using the streaming sample from your jetson.
Hi @JPlou,
As I mentioned in the first posts, we used the UE5 Plugin for video streaming before, but the system delay was 10 seconds in the best situation. Therefore, we’ve started using Live Link on the Jetson to have better performance on video streaming through Live Link.
As a result, please kindly let me know how I can run remote video streaming using the Jetson Orin developer kit and Zed X mini camera. Does Live Link bring the remote video streaming without delay or not?
I hope that you can suggest a developed procedure for implementing this task.
Cheers,
Negin
Also,
We’ve operated according to the steps that you explained but this error came up: CAMERA FAILED TO SETUP.
UVC driver has been installed correctly and we tested it in different resolutions.
Is that possible for you or another support team member to be in contact through WhatsApp or Telegram for better communication?
Hi @NeginHeiran,
As I said, you will not be able to receive a video feed using Live Link, sorry your initial demand was lost in the discussion.
To get the images in Unreal out-of-the-box, you have to use our UE5 plugin and use a streaming app on the Jetson device. To get body tracking data only, you can use Live Link.
I want to clarify something, the latency you’re having is not normal in the local network, which is the only supported way of streaming the SDK. Streaming as in “ingesting ZED Data from a ZED on another device”.
Side note, UVC is a USB protocol, it does not affect ZED X management.
Hi @JPlou,
Our issue is video streaming from a Zed X which is connected to a Jetson in one location and receives it in a PC in another location. They are connected through the internet. Simply, the Zed X must work as a webcam.
Our solution for this issue is using WebRTC, so what I needed to know is does the Zed X is capable of working as a webcam. And, do you know about using WebRTC with Zed X?
Regards,
Negin
@NeginHeiran
You opened another topic with this question, I’ll link it here: Using Zed X as webcam using WebRTC
If you wish to continue the discussion regarding the UE5 plugin or Live Link, please do it here.