Accessing ZED camera frames before SDK processing for low-latency streaming (raw NV12 / async retrieval alternative?)

Software C++
, ,
1

Hi Stereolabs team,

We’re currently using a ZED camera for positional tracking alongside a low-latency video feed for an RC controller / teleoperation workflow.

For the video path, we’re using the Raw Buffers API to retrieve raw NV12 buffers and stream them over GStreamer for low-latency transport. Our current flow is roughly:

zed.grab(runtime_params);
zed.retrieveImage(raw_buffer);

where retrieveImage(raw_buffer) gives us the raw NV12 NvBufSurface, which we push directly into a GStreamer pipeline.

The issue we’re seeing is that zed.grab() latency appears to vary significantly depending on lighting conditions / scene complexity. Under some conditions, grab() becomes noticeably slower, introducing up to ~300 ms of delay in the RC video feed.

We are still using positional tracking, so we understand some SDK processing is required. However, our main concern is the video path latency, and it seems like delays inside grab() are affecting when the raw image buffer becomes available.

Our goal is to retrieve frames for streaming as early as possible, ideally before non-essential SDK image processing occurs, while still allowing positional tracking to run in parallel.

Specifically, we’re wondering if there is currently a way to:

  • Access raw or minimally processed frames (preferably NV12) with lower latency

  • Decouple frame retrieval from heavier SDK processing inside grab()

  • Retrieve frames asynchronously while positional tracking / other SDK processing continues in parallel

I found references to the deprecated async_image_retrieval functionality, which sounds very close to what we’re trying to achieve. Is there a current equivalent approach in the SDK, or another recommended method for low-latency frame access?

Alternatively, is there any plan to expose a lower-level frame access path or reintroduce similar functionality for latency-sensitive applications like teleoperation / RC video streaming?

Thanks!

2

Hi @ShashankVSS
The feature you described is already available by using the read and grab API functions.

You can call read when you only need the image buffer; then you can call grab when you need the ZED SDK to process all the data.

Read more in the API documentation:

3

Thanks, that helps clarify the intended use of read() vs grab().

One thing I’m still not understanding is how this helps maintain a high-rate video stream if grab() remains blocking.

For example, if I do:

zed.read();
zed.retrieveImage(raw_buffer);   // stream frame immediately

zed.grab(runtime_params);        // tracking/depth processing
zed.getPosition(pose);

then the frame is available earlier for that iteration, but if grab() takes 100-300 ms, I still can’t call read() again until grab() returns. That would still cause me to miss several camera frames and reduce the effective video update rate.

So my question is:

  • Is it supported to call read()/retrieveImage() continuously from one thread while another thread calls grab()/getPosition() on the same sl::Camera instance?

  • Or is there another SDK pattern that allows image acquisition to continue while tracking/depth processing is running?

Our goal is to keep a continuous low-latency video stream while allowing positional tracking to run at a lower rate if necessary.

4

You should call grab only when you need it.

No, this is not multithread-safe behavior.

5

Thanks for confirming.

In that case, if grab() is only called occasionally, does positional tracking only receive visual updates on those grab() calls?

Our concern is that reducing grab() frequency may preserve video rate, but could reduce positional tracking quality because the tracking pipeline is no longer processing every camera frame.

Is there a recommended approach for maintaining high-rate video while preserving positional tracking quality? For example, would the newer Sensors API with pipelined process() be the intended solution for this use case?

6

Yes, Positional Tracking processing is performed when you call “grab”.

7

You can apply what’s described here for the ROS 2 Wrapper.
It applies identically to native SDK applications:

1 Like