I am using a ZED Box Orin NX 16GB with a ZED X Mini Stereo Camera and have some questions regarding updates and streaming.
Currently, my system has the following setup:
JetPack 5.1.1 (L4T 35.3.1)
CUDA 11.4
ZED SDK 4.2.5 (recently updated from 4.0.8)
GStreamer 1.16.3
1. System Updates
Are there any further updates available? Running sudo apt update does not list any updates.
For example:
CUDA 12.2 or 12.6
JetPack 5.1.2 (L4T 35.4.1) or JetPack 6.2 (L4T 36.4)
A newer version of GStreamer
2. Streaming Differences & Remote Streaming
What are the differences between GStreamer streaming and ZED SDK streaming?
Can both libraries support remote streaming beyond local networks using techniques like port forwarding?
3. Stereo Video Streaming to Oculus Quest 3
I am currently receiving GStreamer streaming video on Oculus Quest 3. Since I am still exploring options, I am using a VR player app to view the video in SBS (or OU) format for stereoscopic 3D perception.
Would it be possible to develop a Unity or Unreal Engine app on Quest 3 that directly receives the stereo video stream from GStreamer or ZED SDK and correctly displays each image to the corresponding eye?
It looks like features such as ZED_Rig_Stereo might be relevant, but how complex would this development be?
Setup Overview
ZED Camera → ZED Box → Windows PC → Unity App → Oculus Quest 3
If you have any tips or best practices for achieving real-time stereo video streaming in this setup, I would greatly appreciate them.
Is there a specific reason why you wish to update to this Jetpack version?
Our GStreamer wrapper (https://github.com/stereolabs/zed-gstreamer) allows you to capture the RGB images of the ZED Camera and push them in a GStreamer pipeline. Thanks to that, the receiver does not have a dependency on the ZED SDK to be able to retrieve and read the video stream.
With the ZED Streaming module, the SDK is necessary on both ends. The main advantage is a streamed camera can be directly used as input by the ZED SDK, in the exact same way as having a camera plugged into your computer in USB.
I think you can do it by using a VPN.
Our Unity plugin already allows you to display the left and right images of the ZED in a VR headset, you don’t need any additional development.
However, it requires the ZED SDK to run which means the Unity application can not be installed on the headset but only on a Windows or Linus computer and having the headset constantly connected to it.
I am considering updating Jetpack to a newer version for potential performance improvements and additional features. (If there is no performance difference, an update may not be necessary.)
Currently, I have updated ZED SDK to 4.2.5 with Jetpack 5.1.1 (L4T 35.3). I would like to know whether it is possible to update not only the SDK but also CUDA and Jetpack. If so, what would be the correct order and method for doing so?
I understand the differences between GStreamer and ZED SDK for streaming as you mentioned.
If I receive the stream on a Windows 10 system using ZED SDK, would it offer better performance compared to using GStreamer?
Additionally, as you mentioned, I plan to develop a Unity application using the ZED plugin for use with Quest 3. In this case, I’d like to hear your thoughts on the differences between receiving the stream via GStreamer versus using ZED SDK (as if connected via USB).
Lastly, if you have any experience or tips regarding long-distance streaming using RTSP or TCP or recommendations, I would appreciate your insights.
If you don’t need to install your application on the headset itself, I would recommend using the streaming from the ZED SDK. That way, you can take advantage of our Unity plugin and have the image directly displayed in the headset using our prefabs/scripts without the need of re-implementing it.
From my understanding, Updating Zetpack or CUDA is not necessary to achieve better performance.
So, If I send remote streaming videos via RTSP or TCP and receive them using ZED SDK (installed on both PCs-ZED Box orin and Windows), I can directly display the received videos on Quest3 without implemening an application. The Unity plugin, with your prefabs/scripts, simply streams the stereo video, functioning like HDMI connection b/w Quest3 and the receiver PC.