Delay in streaming with Live Link with Unreal Engine using Zed X mini

Hi everyone,

We tried to get video stream with Zed X mini using Live Link. But, there’s about 1 minute delay in streaming. How can we increase the performance and reduce the delay? The delay must be under 1 second in our project.
(We are using NVIDIA Jetson Orin 64GB developer kit.)

Cheers.

Hi, welcome to the community!

One minute seems crazy. You should have less than a second. something like 200ms is the most common situation. Can you explain in details how you are trying to do this?

Hi @alassagne,

Thank you for your response. Apologies for the confusion in my previous message. Our video streaming setup did not involve Live Link with a 1-minute delay as initially stated; instead, it relied on specific pieces of code. To enhance performance, we have now transitioned to using Live Link.

In our project, we’re dealing with remote video streaming from the Zed X mini camera, rather than local streaming. Following the guidelines outlined in the documentation (Live Link for UE5 - Stereolabs), we attempted to implement the JSON file with the provided code snippet:

{
“InitParameters”:
{
“input”: “STREAM”,
“input_path”: 192.168.1.170,
“resolution”: “HD1080”,
“fps”: 60,
“depth_mode”: “ULTRA”
},
“PositionalTrackingParameters”:
{
“enable_pose_smoothing” : true,
“enable_area_memory”: false
},
“BodyTrackingParameters”:
{
“enable_module”: false,
“detection_model”: “HUMAN_BODY_ACCURATE”,
“body_format”: “BODY_38”,
“body_selection”: “FULL”,
“confidence”: 40,
“max_range”: -1,
“minimum_keypoints_threshold” : -1,
“skeleton_smoothing”: 0.2
}
}

However, upon execution, we encountered the following error message: “FAILURE: Standard code for unsuccessful behavior.”

We think that the issue may lie with the JSON file. Given the scarcity of documentation on this particular problem, we seek assistance in understanding how to properly configure the initialization parameters within the JSON file to successfully access the camera.

Any guidance or insights you could provide would be greatly appreciated. Thank you for your attention to this matter.

Hi @NeginHeiran,

You’re right, the doc is lacking on this matter, we’ll work on this.

  • Are you able to see the camera in ZED Depth Viewer via the streaming?
  • The only error I see in your file is that IP in the input_path should be between quotes, else it won’t be able to parse the JSON correctly.

Hi @JPlou,

So, is that possible to use Live Link remotely for now?

  1. we see the camera in Zed Depth Viewer via streaming.
  2. The input_path is ok in the code and the system recognizes the IP, but we receive a FAILURE error after that. (I forgot the quotes here.)

Please kindly let me know how I can use Zed X remotely if Live Link has issues with it currently.

Best.

@NeginHeiran
I do reproduce the issue here, I think.

  • Can you please send me the full log from the sender?

I will look into this as soon as possible and update the sample if needed, there is no reason it should not work.

Hi @JPlou,

Thank you for the follow-up. Here’s the log note from sender:

Created: Sat Feb 17 2024

###########################################
############ DMESG [ZEDX] LOG #############
###########################################

[ 20.213700] zedx 30-0018: Driver Version : v0.5.4
[ 20.213709] zedx 30-0018: Probing v4l2 sensor.
[ 20.213958] zedx 30-0018: tegracam sensor driver:video0_v2.0.6
[ 27.289193] zedx 30-0018: zedx_links_check: 1 link(s) detected
[ 30.212290] zedx 30-0018: Single ZED-X camera detect success
[ 30.440766] zedx 30-0018: zedx_set_frame_rate: Unsupported value
[ 30.769063] tegra-camrtc-capture-vi tegra-capture-vi: subdev zedx 30-0018 bound
[ 30.770565] zedx 30-0018: Detected ZED-X sensor
[ 30.770847] zedx 30-0010: Driver Version : v0.5.4
[ 30.770852] zedx 30-0010: Probing v4l2 sensor.
[ 30.771109] zedx 30-0010: tegracam sensor driver:video1_v2.0.6
[ 30.785123] zedx 30-0018: zedx_open: Accessing the camera
[ 30.877412] zedx 30-0010: zedx_links_check: 1 link(s) detected
[ 31.102952] zedx 30-0010: zedx_set_frame_rate: Unsupported value
[ 31.432660] tegra-camrtc-capture-vi tegra-capture-vi: subdev zedx 30-0010 bound
[ 31.433791] zedx 30-0010: Detected ZED-X sensor
[ 31.433951] zedx 31-0018: Driver Version : v0.5.4
[ 31.433953] zedx 31-0018: Probing v4l2 sensor.
[ 31.434117] zedx 31-0018: tegracam sensor driver:video2_v2.0.6
[ 31.437164] zedx 30-0010: zedx_open: Accessing the camera
[ 31.541393] zedx 31-0018: zedx_links_check: 1 link(s) detected
[ 31.542083] zedx 31-0018: zedx_write_reg:i2c write failed, 0x301a = 2058
[ 31.549050] zedx 31-0018: ZED-X detect error
[ 31.549186] zedx: probe of 31-0018 failed with error -1
[ 31.554668] zedx 31-0010: Driver Version : v0.5.4
[ 31.554672] zedx 31-0010: Probing v4l2 sensor.
[ 31.554971] zedx 31-0010: tegracam sensor driver:video3_v2.0.6
[ 31.661472] zedx 31-0010: zedx_links_check: 1 link(s) detected
[ 31.662067] zedx 31-0010: zedx_write_reg:i2c write failed, 0x301a = 2058
[ 31.669062] zedx 31-0010: ZED-X detect error
[ 31.669197] zedx: probe of 31-0010 failed with error -1
[ 31.981098] zedx 30-0010: zedx_open: Accessing the camera
[ 31.981174] zedx 30-0018: zedx_open: Accessing the camera
[95471.781559] zedx 30-0010: zedx_open: Accessing the camera
[95471.785278] zedx 30-0018: zedx_open: Accessing the camera

###########################################
############ DMESG [DTS] LOG #############
###########################################

[ 0.004259] DTS File Name: /home/louis/Documents/jetson/QA/35.3/jetson-gmsl-kernel/src/kernel/kernel-5.10/arch/arm64/boot/dts/…/…/…/…/…/…/hardware/nvidia/platform/t23x/concord/kernel-dts/tegra234-p3701-0000-p3737-0000.dts
[ 2.504698] tegra-pmc c360000.pmc: scratch reg offset dts data not present

###########################################
############ DTB list original ###########
###########################################

/boot/dtb/kernel_tegra234-p3701-0000-p3737-0000.dtb

###########################################
############ DTB list SLabs ###########
###########################################

/boot/stereolabs/tegra194-p2888-0001-p2822-0000.dtb
/boot/stereolabs/tegra194-p3668-0000-p3509-0000.dtb
/boot/stereolabs/tegra194-p3668-0001-p3509-0000.dtb
/boot/stereolabs/tegra234-p3701-0000-p3737-0000.dtb
/boot/stereolabs/tegra234-p3767-0000-p3509-a02.dtb
/boot/stereolabs/tegra234-p3767-0000-p3768-0000-a0.dtb
/boot/stereolabs/tegra234-p3767-0001-p3509-a02.dtb
/boot/stereolabs/tegra234-p3767-0003-p3768-0000-a0.dtb

/boot/stereolabs/utils:
extlinux.py
fio.py
pycache

###########################################
############ EXTlinux file ###########
###########################################

TIMEOUT 30
DEFAULT Stereolabs

MENU TITLE L4T boot options

LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
FDT /boot/dtb/kernel_tegra234-p3701-0000-p3737-0000.dtb
INITRD /boot/initrd
APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 console=ttyAMA0,115200 console=tty0 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0 nv-auto-config

When testing a custom kernel, it is recommended that you create a backup of

the original kernel and add a new entry to this file so that the device can

fallback to the original kernel. To do this:

1, Make a backup of the original kernel

sudo cp /boot/Image /boot/Image.backup

2, Copy your custom kernel into /boot/Image

3, Uncomment below menu setting lines for the original kernel

4, Reboot

LABEL backup

MENU LABEL backup kernel

LINUX /boot/Image.backup

FDT /boot/dtb/kernel_tegra234-p3701-0000-p3737-0000.dtb

INITRD /boot/initrd

APPEND ${cbootargs}

LABEL Stereolabs
MENU LABEL Stereolabs kernel
LINUX /boot/Image
FDT /boot/stereolabs/tegra234-p3701-0000-p3737-0000.dtb
INITRD /boot/initrd
APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 console=ttyAMA0,115200 console=tty0 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0 nv-auto-config

####################################
############ Driver list ###########
####################################

/usr/lib/modules/5.10.104-tegra/kernel/drivers/stereolabs/max96712/max96712.ko==> exists
/usr/lib/modules/5.10.104-tegra/kernel/drivers/stereolabs/zedx/sl_zedx.ko ==> exists

################################################
############ ZEDX Daemon Status LS #############
################################################

● zed_x_daemon.service - ZED-X Daemon service
Loaded: loaded (/etc/systemd/system/zed_x_daemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2024-02-12 05:57:45 EST; 5 days ago
Main PID: 1186 (ZEDX_Daemon)
Tasks: 3 (limit: 74725)
Memory: 18.5M
CGroup: /system.slice/zed_x_daemon.service
└─1186 /usr/sbin/ZEDX_Daemon

Warning: journal has been rotated since unit was started, output may be incomplete.

################################################
############ I2C Trace [-l] #############
################################################

i2c-3 i2c 3190000.i2c I2C adapter
i2c-30 i2c i2c-2-mux (chan_id 0) I2C adapter
i2c-1 i2c c240000.i2c I2C adapter
i2c-8 i2c 31e0000.i2c I2C adapter
i2c-6 i2c 31c0000.i2c I2C adapter
i2c-33 i2c i2c-2-mux (chan_id 3) I2C adapter
i2c-4 i2c Tegra BPMP I2C adapter I2C adapter
i2c-31 i2c i2c-2-mux (chan_id 1) I2C adapter
i2c-2 i2c 3180000.i2c I2C adapter
i2c-0 i2c 3160000.i2c I2C adapter
i2c-9 i2c NVIDIA SOC i2c adapter 0 I2C adapter
i2c-7 i2c c250000.i2c I2C adapter
i2c-5 i2c 31b0000.i2c I2C adapter
i2c-32 i2c i2c-2-mux (chan_id 2) I2C adapter

################################################
############ I2C Trace /dev/i2C-30 #############
################################################

 0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f

00: – – – – – – – – – – – – –
10: UU – – – – – – – UU 19 – – – – – –
20: – – – – – – – – – UU – – – – – –
30: – – – – – – – – – – – – – – – –
40: – – – – – – – – – 49 – – – – – –
50: – – – – 54 55 – – 58 59 – – – – – –
60: – – 62 – – – – – – 69 – – – – – –
70: UU – – – – – – –

################################################
############ I2C Trace /dev/i2C-31 #############
################################################

 0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f

00: – – – – – – – – – – – – –
10: – – – – – – – – – – – – – – – –
20: – – – – – – – – – 29 – – – – – –
30: – – – – – – – – – – – – – – – –
40: – – – – – – – – – – – – – – – –
50: – – – – – – – – – – – – – – – –
60: – – – – – – – – – – – – – – – –
70: UU – – – – – – –

################################################
############ I2C Trace /dev/i2C-32 #############
################################################

 0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f

00: – – – – – – – – – – – – –
10: – – – – – – – – – – – – – – – –
20: – – – – – – – – – – – – – – – –
30: – – – – – – – – – – – – – – – –
40: – – – – – – – – – – – – – – – –
50: – – – – – – – – – – – – – – – –
60: – – – – – – – – – – – – – – – –
70: UU – – – – – – –

################################################
############ I2C Trace /dev/i2C-33 #############
################################################

 0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f

00: – – – – – – – – – – – – –
10: – – – – – – – – – – – – – – – –
20: – – – – – – – – – – – – – – – –
30: – – – – – – – – – – – – – – – –
40: – – – – – – – – – – – – – – – –
50: – – – – – – – – – – – – – – – –
60: – – – – – – – – – – – – – – – –
70: UU – – – – – – –
#########################

@NeginHeiran
Thanks for the logs!

It seems like the issue is before Live Link, in the sender on the jetson.
What resolution/fps are you trying to stream?

zedx_set_frame_rate: Unsupported value

This line indicates an unsupported setting in the sender. At the core, there may be a fallback value being used, creating a mismatch with the expected value in Live Link.

Dear @JPlou,

Thank you too!
We used 60fps, for your reference I attached the part of the code about resolution.

@NeginHeiran

Can you confirm that your setup is:

  • Jetson with an SDK streaming sample, streaming the feed of the ZED X Mini
  • Windows computer with the Live Link sender, receiving STREAM data from the jetson
  • UE5 Live Link sample, latest release

Is that correct?

Then, some questions:

  • Which versions of the SDK are you using on the jetson and computer?
  • Which version of the plugin are you using on the computer?
  • In ZED Explorer on your main computer, what resolution/fps do you get when streaming the ZED X mini on the jetson?
  • You are able to see depth data in Depth Viewer from the feed streamed from the jetson on your computer, right?
  • Are you using the default streaming sample from the SDK in Python on your jetson? If you made any modifications to it, can you include them here?
  • Is the FAILURE error you get from the Live Link sender or the streaming sender on the jetson?
  • Can you send the log from the Live Link sender here?

Sorry, that’s a bunch of questions, but I hope they’ll help pinpoint the issue because I don’t get the FAILURE message on my side.

Hi @JPlou,

The zed X mini camera is connected to the Jetson which is listening so that a windows or a PC connect to it.
On the other side, there’s a windows PC which want to connect to the Jetson through live link.
Thus, I think I can confirm what you said at first.

On both systems (windows and Jetson) SDK 4 has been installed.

Both devices have 60fps and 1080 resolution.

We are using Python streaming sample code on Jetson.

We have a failure error on the Windows side when pc wants to connect to the Jetson through Live Link. I attached the screenshot of the error and settings. The settings in Windows and Jetson are the same.

Cheers,
Negin

@NeginHeiran

To be sure:

  • Are you able to use 1080p 30fps?
  • Are you specifying the port (like so: input_path:“192.168.1.160:30000” with the port you use in the Python sender) in the configuration file?

Also!

I missed your last answer in your other post, that could have helped you I think, sorry.

While the Live Link sender is currently not compatible with Jetson, we’ve developed a version that works independently of the Unreal source files and thus can be used directly on Jetson.
It’s currently on this branch on the repo, not documented yet, and not guaranteed bug-free yet, but it can be worth a try in your case I think.