Low frame rate of ZED X on Jetson AGX Orin

d1324 · January 11, 2025, 8:54pm

Hey, we’re running the zed-ros2-wrapper with ROS2 Humble on Jetson AGX Orin (JetPack version 6.1, SDK version 4.2).

The frame rate grab_frame_rate is the default value of 30 (confirmed with rosparam get), yet running ros2 topic hz /zed/zed_node/rgb/image_rect_color runs at barely 14Hz.

Couple more notes:

Running ROS2 nodes are:

/zed/transform_listener_impl_ffff5c09ae30
/zed/zed_container
/zed/zed_node
/zed/zed_state_publisher

Resolution grab_resolution: 'HD1080'
Turning off depth (depth_mode: "NONE") doesn’t help
jetson_clocks is running
jtop shows CPU ~10%, GPU 0~4%
Power mode MAXN

ros2 topic hz /zed/zed_node/rgb/image_rect_color output:

average rate: 13.149
	min: 0.067s max: 0.134s std dev: 0.02276s window: 15
average rate: 13.978
	min: 0.066s max: 0.134s std dev: 0.01672s window: 30
average rate: 14.276
	min: 0.066s max: 0.134s std dev: 0.01381s window: 45
average rate: 13.466
	min: 0.066s max: 0.201s std dev: 0.02429s window: 56

ros2 topic bw /zed/zed_node/rgb/image_rect_color output:

133.11 MB/s from 14 messages
	Message size mean: 8.29 MB min: 8.29 MB max: 8.29 MB
123.67 MB/s from 28 messages
	Message size mean: 8.29 MB min: 8.29 MB max: 8.29 MB
121.30 MB/s from 42 messages
	Message size mean: 8.29 MB min: 8.29 MB max: 8.29 MB
119.96 MB/s from 56 messages
	Message size mean: 8.29 MB min: 8.29 MB max: 8.29 MB

jetson_clocks --show output:

SOC family:tegra234  Machine:NVIDIA Jetson AGX Orin Developer Kit
Online CPUs: 0-11
cpu0:  Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu1:  Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu2:  Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu3:  Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu4:  Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu5:  Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu6:  Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu7:  Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu8:  Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu9:  Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu10: Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu11: Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
GPU MinFreq=1300500000 MaxFreq=1300500000 CurrentFreq=1300500000
Active GPU TPCs: 8
EMC MinFreq=204000000 MaxFreq=3199000000 CurrentFreq=3199000000 FreqOverride=1
DLA0_CORE:   Online=1 MinFreq=0 MaxFreq=1600000000 CurrentFreq=1600000000
DLA0_FALCON: Online=1 MinFreq=0 MaxFreq=844800000 CurrentFreq=844800000
DLA1_CORE:   Online=1 MinFreq=0 MaxFreq=1600000000 CurrentFreq=1600000000
DLA1_FALCON: Online=1 MinFreq=0 MaxFreq=844800000 CurrentFreq=844800000
PVA0_VPS0: Online=1 MinFreq=0 MaxFreq=1369600000 CurrentFreq=1369600000
PVA0_AXI:  Online=1 MinFreq=0 MaxFreq=985600000 CurrentFreq=985600000
FAN Dynamic Speed Control=nvfancontrol hwmon0_pwm1=53
NV Power Mode: MAXN

How can we increase the frame rate?

ba2sakal · January 13, 2025, 9:05am

I have the exact same problem even though I use 4 cameras the, GPU is not utilized somehow, stay around 0-20%.

I am afraid it is because of the NVArgus update that came with 6.1.

JetPack 6.1 Release Notes — JetPack 6.1 documentation

Camera

Enhanced the Argus library, reducing CPU utilization by up to 40%.

And somehow it gets the power from CPU. On the other hand the images I get from cameras are bad quality and scale of the objects in the image is distorted.

Myzhar · January 13, 2025, 10:39am

Hi @ba2sakal and @d1324
did you follow this guide to improve ROS 2 performance?

ba2sakal · January 13, 2025, 10:47am

@d1324
After reactivating depth mode the GPU utilization is normal 60-100. With Depth Mode: None, GPU is utilized as you said 0-4% and frame rate is 30, here is my result:
#149 - Freq: 30.01 Hz (Avg: 30.06 Hz) - Bandwidth: 1899.10 Mbps (Avg: 1902.07 Mbps) - Msg size:

Also I would suggest to use

ros2 run zed_topic_benchmark zed_topic_benchmark --ros-args -p topic_name:=“$topic”

to test topics.

On the other hand the FPS drops if you try to visualize your images in rviz or rqt at the same time. If you do it that way.

Myzhar · January 13, 2025, 11:07am

Yes, it’s not a good practice to run rviz2 or rqt on the Jetson because they take a lot of resources.

This is correct.

Are you using a “custom” output resolution?

github.com

stereolabs/zed-ros2-wrapper/blob/master/zed_wrapper/config/common_stereo.yaml#L24-L25


      
          pub_resolution: 'CUSTOM' # The resolution used for image and depth map publishing. 'NATIVE' to use the same `general.grab_resolution` - `CUSTOM` to apply the `general.pub_downscale_factor` downscale factory to reduce bandwidth in transmission
          pub_downscale_factor: 2.0 # rescale factor used to rescale image before publishing when 'pub_resolution' is 'CUSTOM'

Myzhar · January 13, 2025, 1:46pm

Right, you said before that you are using JP6.1 now.
Unfortunately, this is a known problem with JP6.1 and we are working with NVIDIA to solve it as quickly as possible.

ba2sakal · January 13, 2025, 1:56pm

So it is not the problem of my setup but it is a general problem. I was thinking I made a mistake while setting up the JP6.1 configuration.
Do you think this low resolution would affect other algorithms such as VSLAM, object detection etc ?

Thank you for your time!

ba2sakal · January 13, 2025, 2:13pm

@Myzhar
Since JetPack 6.1 was released several months ago and this problem persists, do you think it will take much longer to resolve? I have a project to complete soon and need reliable results. Would you recommend rolling back to JetPack 6.0, or do you think a fix for this issue will be available soon?

I was not able to find similar topic online, I guess there is no work around?

Myzhar · January 13, 2025, 2:34pm

As I told you we are working on a fix. It’s a sneaky problem, otherwise, the fix was released more quickly.

It seems that the cause of this problem is the de-noise filter that is not working the same way it was before JP6.1.
You can first try to disable the denoise control by setting the DENOISING camera control value to 0.
If the result does not satisfy your requirements, a rollback to JP6.0 can be the best way to take until we release a fix.

ba2sakal · January 13, 2025, 3:33pm

Thanks a lot you for your time and effort in providing a detailed response. I will try the suggested solution.

d1324 · January 15, 2025, 7:42pm

Hi @Myzhar, we had not initially but are now. We set up Cyclone DDS and changed the settings accordingly.

The frame rate remains to be around 15Hz as measured by: ros2 run zed_topic_benchmark zed_topic_benchmark --ros-args -p topic_name:=/zed/zed_node/rgb/image_rect_color:

#142 - Freq: 15.00 Hz (Avg: 15.10 Hz) - Bandwidth: 949.21 Mbps (Avg: 955.34 Mbps) - Msg size: 7.91 MB

Changing DEPTH_MODE did not help. Still running jetson_clocks

BTW this is not using any visualization tools, no rviz2 or rqt are running. We’re benchmarking by running the ZED node only!

EDIT 1: My bad, was looking at the wrong config field. The publish frequency is under pub_frame_rate instead of grab_frame_rate. Changing pub_frame_rate: 30 works. Problem solved, thank you.