I have three ZedX Minis connected to a NVIDIA Jetson AGX Orin. I am using this script to stream the feed to my host machine.
My jetson has the following config:
Jetpack v6.2
QuadLink driver v1.3.0
Zed SDK v5.0.3
Often, when we grab a frame, I see that the error code returned is CAMERA REBOOTING and it reboots usually after 10-30s. I see that it is able to successfully reboot but we cant use any of the stream to retrieve data during this reboot process and have to wait until the error code returned is SUCCESS. Is there a fix for this?
Here is the log that I see on the streaming side.
Warning: possible framedrop, time between frames was 24ms
Warning: possible framedrop, time between frames was 26ms
Warning: possible framedrop, time between frames was 27ms
Warning: possible framedrop, time between frames was 24ms
Warning: possible framedrop, time between frames was 23ms
Warning: possible framedrop, time between frames was 23ms
Warning: possible framedrop, time between frames was 25ms
Warning: possible framedrop, time between frames was 26ms
Warning: possible framedrop, time between frames was 26ms
Warning: possible framedrop, time between frames was 33ms
Warning: possible framedrop, time between frames was 24ms
Warning: possible framedrop, time between frames was 24ms
Warning: possible framedrop, time between frames was 26ms
Warning: possible framedrop, time between frames was 59ms
Grab error: CAMERA REBOOTING
Warning: possible framedrop, time between frames was 10847ms
Grab error: CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
Grab error: CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
Grab error: CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
Grab error: CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
Grab error: CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
Grab error: CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
Grab error: CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
Grab error: CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
It seems that you are overloading the Jetson, and this could cause the ZED X Driver to timeout and issue a camera reboot to recover.
Please add more information concerning your setup:
The resolutions we are using are HD1200@60fps for all cameras and jetson power mode is set to MAXN. We have set jetson clocks are well.
The jetson cpu usage per core is around 85%, gpu usage is around 40% for the most part occasionally going upto 80% and falling back. The temperatures seem to be round about 55-60 degrees celcius.
Let me know if you need any other information. We have faced similar issue in the past where the streaming would seg fault after a while as we shared in this forum post.
But now the streaming does not raise a segmentation fault but the cameras reboot successfully after a while. Below, I am sharing the message from the forum post where we faced the same issue.
We quite often observe the following error during streaming but the streaming again runs fine after a while and the jetson ram,cpu,gpu,temperature everything looks normal. One additional thing to note here is we have been monitoring the time interval between 2 consecutive frames and the error below comes when the time between 2 consecutive frames is ~10000ms. I have shared a link to all the logs at the bottom of this post
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error InvalidState: Argus client is exiting with 11 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 366)
(Argus) Error EndOfFile: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 277)
(Argus) Error EndOfFile: Receive worker failure, notifying 11 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 350)
(Argus) Error InvalidState: Argus client is exiting with 11 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 366)
(Argus) Error EndOfFile: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 379)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)