Zed X SVO recording drops many frames

Software ROS2
1

Sorry if this has been covered elsewhere. I looked around and did not see anyone with this issue.

A little background. I am using 4 Zed X with Jetson Orin AGX 64. When I first set this system up, the Jetson was unable to handle all 4 cameras in ROS/ROS2 reliably. I ended up setting the Jetson up as streamer, and then processed the stream on a larger GPU (A6000) on the same vehicle. This worked pretty reliably, but even the A6000 seemed to struggle if I ran all 4 cameras in ROS at anything higher than NEURAL_LIGHT.
Fast forward a bit, and things got remarkably better with SDK updates. 5.1+ has been a lot better on the big computer. But I was dropping many frames when recording SVOs. I thought this might be because it gets compressed to stream, then decompressed on the host then recompressed to record (though I am not sure if this is correct).

I decided to try setting up the Jetson again with updated SDK to see how it did, thinking if it could handle compressing for streaming it should be able to handle compressing for SVO. I I have fresh JP6.2/ROS2 Humble install. I had to set the ROS2 wrapper settings very “light weight” in order to get this to work. I set resolution to HD1200, capture rate to 30 FPS, and NEURAL_LIGHT. I set all the publishing frequencies to < 1Hz. I set the compression to Lossy H265. I recorded a few minutes like this and had no dropped frames. I went out and drove the vehicle and recorded again. Suddenly I have tons of dropped frames. I had an effective frame rate in the SVO < 2Hz. I attribute this to maybe the compression can do some caching when the vehicle is stationary that it can not do when in motion? But this seems excessive.

On top of that, it seems I can not open the cameras at all using HD1080 and they often fail to open in SVGA. I feel like I am doing something really wrong.

Is there a better way to handle recording through ROS2? I can look into writing a simple recording only script, but integration with ROS is nice to consolidate the recording calls to a single location. (I am using this data with Lidar/GNSS/etc… all working in ROS).

What would cause the cameras to not open when using resolutions other than HD1200?

Should a Jetson Orin be able to stream 4 cameras reliably or am I asking for too much from the computer?

When I call record I set:

bitrate: 0

target_fps: 0

compression_mode: 0

input_transcode: false

Which should choose appropriate defaults I believe? My zed config file and a terminal log launch using HD1200 followed by a failed launch using HD1080 are also attached.

launch.log (274.7 KB)

One last thing. I should mention that this is custom carrier board (NRU-230) but the drivers do not seem to be an issue, at least getting connected to the cameras initially.

zed_record.yaml (23.5 KB)

Thank you!

Benjamin

2

Hi @benijohn
What GMSL2 capture card are you using?

Have you applied the configurations described in the ROS 2 documentation?

3

@Myzhar Sorry for the delay here.

The capture card is whatever is used in Neousys NRU-230. I can try to find out what exactly they are using.

I am curious though, there are seemingly two issues here:

The frame drop is a big one and this happens no matter if I record on Jetson natively or record the stream. We only use ROS to call the service so ROS network should not be an issue. Or am I missing something with how recording happens? Does the stream have to go through ROS middleware before being dumped to SVO?

The second issue with cameras not opening in resolutions seems to be a low level issue. I can open them all individually in any resolution, but as soon as I call my multi-camera launch with resolution < 1200HD I get some weird nvargus error and the nodes crash. I will try to get a trace of the exact failure.

But other than using ROS to call the recording service, shouldn’t that actual recording happen at a lower level than ROS?

Thank you!

Benjamin

4

I just want to add that I have tried using the Zed tutorial multi-camera recording example with the same result. Indoors/static, we get clean 30 fps recordings. As soon as the vehicle starts moving, our frame rate drops to about 4-5 fps, with sometimes > 1 second between frames.

I found another post where the users experienced dropped frames due to auto exposure settings, but disabling auto exposure does not seem to help.

We have set power mode to MAXN, we use Jetson Clocks. We have even tried a Gstreamer pipeline, not matter what we do as soon as the vehicle begins moving the frame rate becomes unusable.

Does this point to computer issue? Driver issue? We are on driver 1.3 (the latest from Neousys), I have reached out to them to find out when we will get the updated version, but I am curious if there is anything I can do in the mean time to get the frame rates up. Or debug why this happens?

Thank you!

Benjamin

5

Hi,

So this is related to the driver issue. I am not sure why the parameters are not being set properly in ROS, the ROS params say AWB/AE are false, but the cameras still adjust automatically to lighting changes. However, we wrote a custom script using the SDK directly and set AWB/AE false and the cameras do much much better than before. We can now record about 13-14 fps under pretty fast movement. We do occasionally, get corrupted frames, but this seems to be related to vibrations at certain resonate frequencies.

I have reached out to Neousys to request the latest drivers so hopefully that will fully remedy the issue.

I guess now the only remaining question is, should we still be able to get 30 fps as is stated here: Record and Replay camera data with ROS2 - Stereolabs or is this more of a hypothetical number? It would be great to get a few more fps, but as of now this should get us started.

Thank you!

Benjamin