Hi Stereolabs team,
We’re currently using a ZED camera for positional tracking alongside a low-latency video feed for an RC controller / teleoperation workflow.
For the video path, we’re using the Raw Buffers API to retrieve raw NV12 buffers and stream them over GStreamer for low-latency transport. Our current flow is roughly:
zed.grab(runtime_params);
zed.retrieveImage(raw_buffer);
where retrieveImage(raw_buffer) gives us the raw NV12 NvBufSurface, which we push directly into a GStreamer pipeline.
The issue we’re seeing is that zed.grab() latency appears to vary significantly depending on lighting conditions / scene complexity. Under some conditions, grab() becomes noticeably slower, introducing up to ~300 ms of delay in the RC video feed.
We are still using positional tracking, so we understand some SDK processing is required. However, our main concern is the video path latency, and it seems like delays inside grab() are affecting when the raw image buffer becomes available.
Our goal is to retrieve frames for streaming as early as possible, ideally before non-essential SDK image processing occurs, while still allowing positional tracking to run in parallel.
Specifically, we’re wondering if there is currently a way to:
-
Access raw or minimally processed frames (preferably NV12) with lower latency
-
Decouple frame retrieval from heavier SDK processing inside
grab() -
Retrieve frames asynchronously while positional tracking / other SDK processing continues in parallel
I found references to the deprecated async_image_retrieval functionality, which sounds very close to what we’re trying to achieve. Is there a current equivalent approach in the SDK, or another recommended method for low-latency frame access?
Alternatively, is there any plan to expose a lower-level frame access path or reintroduce similar functionality for latency-sensitive applications like teleoperation / RC video streaming?
Thanks!