Streaming ZED X in real time?

Software
,
1

Hello,

My project is to shoot a SBS video using the ZED X camera, stream it in real-time(as fast as possible) and watch it through the Oculus Quest 3 for a 3D effect.

I have a few questions:

  1. which plugins are the fastest for streaming ZED X videos? I’m considering Gstreamer, OpenCV, RTSP, etc.
  2. Has anyone had experienced watching ZED X SBS videos on a HMD? HMD Video players like DeoVR and HereSphere are some examples, but I’m unsure if they are compatible with ZED X videos.

I’m also facing difficulties using the ZED Box Orin NX since I’m new to Linux,
Any advice from someone who has experienced streaming with the ZED camera would be greatly appreciated.

Thank you in advance!

2

Hi @Ethan,

  1. which plugins are the fastest for streaming ZED X videos? I’m considering Gstreamer, OpenCV, RTSP, etc.

I think we don’t have tests on this use case in-house, but we do have GStreamer and OpenCV integrations to get you started.

I’m also facing difficulties using the ZED Box Orin NX since I’m new to Linux,

Don’t hesitate if you have questions! The documentation and samples may also prove helpful to understand all the concepts around the ZED SDK.

That’s an interesting project, I’m very curious of the output, please don’t hesitate to post a show-off topic once you get some results! :wink:

3

Also, this topic mentions streaming to an HMD, you might want to take a look: Stream Video to non-CUDA device - #8 by keli95566

4

Thank you @JPlou ,

I am currently working with GStreamer, and have come across some examples that use RTSP.
However, in [GStreamer - ZED Camera Source - Stereolabs], (GStreamer - ZED Camera Source - Stereolabs)
it mentions using H264 streaming over UDP on a local network,
while the https://www.stereolabs.com/docs/gstreamer/rtsp-server
it uses a ZED RTSP server.

I am wondering if you have any recommendations on which option would be beter for streaming ZED X videos and if it is possible to watch the streamed videos in other places through the ZED RTSP server on a Windows platform (via https:// ?).

Best regards

5

Hey @Ethan,

Sorry, this is a bit out of my expertise, and knowledgeable people in the office are unavailable now.
I’ll have to get back to you next week (or one of my colleagues will), sorry for the delay.

1 Like
6

Hi @Ethan
if you need real-time visualization with no latency I do not recommend using RTSP, but a direct UDP connection.
The configuration to be used for data streaming depends on the capabilities of the Oculus Quest 3. If it supports GStreamer and has hardware H264/H265 decoding capabilities I think this is the best option to chose.

7

Thank you @Myzhar ,for your reply.

After several attemps, I managed to get videos from ZED Box Orin onto my Windows PC with low latency, although the resolution is also bad when using RTSP.

  1. Regarding your recoomendation for using a UDP connection for low latency, Does this mean following the instructions outlined here: [GStreamer - ZED Camera Source - Stereolabs]?

  2. Despite trying various encoders and decoders, the latency is still low, and the frame rates are around 30~35 fps with poor resolution. Would using a UDP improve this? Or do you have any specific encoders/decoders that you have found to work well?

FYI, My codes for the ZED Box Orin are:

gst-zed-rtsp-launch -a {ZED BOX IP ADDRESS} zedsrc stream-type=2 camera-resolution=2 camera-fps=60 depth-mode=0 ! videoconvert ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! **nvv4l2h264enc** isert-sps-pps=1 maxperf-enalbe=1 bitrate=120000 ! rtph24pay config-interval=-1 mtu=15000 pt=96 name=pay0,

and for the Windows PC are:

gst-launch-1.0 rtspsrc location=rtsp://{ZED BOX IP ADREESS} latency=0 ! application/x-rtp, clock-rate=90000, payload=96 ! queue ! rtph264depay ! h264parse ! **avdec_h264** ! videoconvert ! fpsdisplaysink

(Yes I have also tried using decoders like nvh264dec but it seems to add more latency, possibly due to color conversion?)

Anyway, I must say the ZED SDK with GStreamer is excellent, Thank you again for your assistance!

9

There are different options to try to reduce latency.
What option to use depends on the configuration of your system and the versions of the extensions that you are using.

I recommend exploring all the available options by using the command gst-inspect-1.0 with each of the extensions on both the sender and receiver side, for example gst-inspect-1.0 nvv4l2h264enc and gst-inspect-1.0 avdec_h264

Furthermore, the quality of the network can influence the final latency, so the bitrate can play an important role in this aspect.

1 Like
10

Hello, @Myzhar! I had the zed-oculus application working, built according to your example at Stereo Video Passthrough for Oculus Rift | Stereolabs, so that I could view the ZED camera image instead of the through mode. It worked with the old ZED camera and the Oculus Quest 2 headset. Then, I acquired the Oculus Quest 3 and tested the old build. It worked with the new headset, but the window became narrower due to the different display size of the headset. Now, I have purchased the ZED X Mini and ZED Box Orin for the same purposes - broadcasting the stereo stream from the camera to the headset (both locally and remotely). I installed the new ZED SDK on my laptop. The zed-oculus program stopped working. Could you please tell me where I need to make changes in the code to adapt this program to the new devices?