Error code CAMERA REBOOTING when streaming multiple cameras over Jetson

Software
1

Hello,

I have three ZedX Minis connected to a NVIDIA Jetson AGX Orin. I am using this script to stream the feed to my host machine.

My jetson has the following config:
Jetpack v6.2
QuadLink driver v1.3.0
Zed SDK v5.0.3

Often, when we grab a frame, I see that the error code returned is CAMERA REBOOTING and it reboots usually after 10-30s. I see that it is able to successfully reboot but we cant use any of the stream to retrieve data during this reboot process and have to wait until the error code returned is SUCCESS. Is there a fix for this?

Here is the log that I see on the streaming side.

Warning: possible framedrop, time between frames was 24ms
Warning: possible framedrop, time between frames was 26ms
Warning: possible framedrop, time between frames was 27ms
Warning: possible framedrop, time between frames was 24ms
Warning: possible framedrop, time between frames was 23ms
Warning: possible framedrop, time between frames was 23ms
Warning: possible framedrop, time between frames was 25ms
Warning: possible framedrop, time between frames was 26ms
Warning: possible framedrop, time between frames was 26ms
Warning: possible framedrop, time between frames was 33ms
Warning: possible framedrop, time between frames was 24ms
Warning: possible framedrop, time between frames was 24ms
Warning: possible framedrop, time between frames was 26ms
Warning: possible framedrop, time between frames was 59ms
Grab error:  CAMERA REBOOTING
Warning: possible framedrop, time between frames was 10847ms
Grab error:  CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
Grab error:  CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
Grab error:  CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
Grab error:  CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
Grab error:  CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
Grab error:  CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
Grab error:  CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
Grab error:  CAMERA REBOOTING
Warning: possible framedrop, time between frames was 34ms
2

Hi @abhishekpavani

It seems that you are overloading the Jetson, and this could cause the ZED X Driver to timeout and issue a camera reboot to recover.
Please add more information concerning your setup:

  • resolutions
  • frame rates
  • Jetson power mode
  • other useful info
3

Hi @Myzhar ,

The resolutions we are using are HD1200@60fps for all cameras and jetson power mode is set to MAXN. We have set jetson clocks are well.

The jetson cpu usage per core is around 85%, gpu usage is around 40% for the most part occasionally going upto 80% and falling back. The temperatures seem to be round about 55-60 degrees celcius.

Let me know if you need any other information. We have faced similar issue in the past where the streaming would seg fault after a while as we shared in this forum post.

But now the streaming does not raise a segmentation fault but the cameras reboot successfully after a while. Below, I am sharing the message from the forum post where we faced the same issue.

  1. We quite often observe the following error during streaming but the streaming again runs fine after a while and the jetson ram,cpu,gpu,temperature everything looks normal. One additional thing to note here is we have been monitoring the time interval between 2 consecutive frames and the error below comes when the time between 2 consecutive frames is ~10000ms. I have shared a link to all the logs at the bottom of this post
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error InvalidState: Argus client is exiting with 11 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 366)
(Argus) Error EndOfFile: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 277)
(Argus) Error EndOfFile: Receive worker failure, notifying 11 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 350)
(Argus) Error InvalidState: Argus client is exiting with 11 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 366)
(Argus) Error EndOfFile: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 379)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
4

Please try reducing to 30 FPS.

5

Hi @Myzhar ,

As you suggested, I ran the experiment again with the HD1200@30fps settings and still found the cameras to be rebooting multiple times. I monitor the jetson stats and have logged everything. You can find the logs here

I am sharing a summary of when the camera reboot happens across each of the 3 cameras and when it reboots back. It takes about 30-40s for it reboot back

cam serial: 54832066:
**1st time cam reboot** : Tuesday, July 22, 2025 2:32:04.656 PM, reboots back at Tuesday, July 22, 2025 2:32:39.477 PM
2: Tuesday, July 22, 2025 3:05:08.081 PM, reboots back at Tuesday, July 22, 2025 3:05:42.733 PM
3: Tuesday, July 22, 2025 11:56:05.685 PM, reboots back at Tuesday, July 22, 2025 11:56:40.783 PM
4: Wednesday, July 23, 2025 3:13:01.280 AM, reboots back at Wednesday, July 23, 2025 3:13:36.807 AM

cam serial: 57366562:
1st time error: Tuesday, July 22, 2025 2:32:04.660 PM, reboots back at Tuesday, July 22, 2025 2:32:36.423 PM
2: Tuesday, July 22, 2025 3:05:08.073 PM, reboots back at Tuesday, July 22, 2025 3:05:34.698 PM
3. Tuesday, July 22, 2025 11:56:05.799 PM reboots back at Tuesday, July 22, 2025 11:56:37.515 PM
4. Wednesday, July 23, 2025 3:13:01.272 AM, reboots back at Wednesday, July 23, 2025 3:13:33.430 AM

cam serial: 57401118:
1: Tuesday, July 22, 2025 2:32:04.116 PM, reboots back at Tuesday, July 22, 2025 2:32:44.400 PM
2. Tuesday, July 22, 2025 3:05:08.076 PM, reboots at Tuesday, July 22, 2025 3:05:39.208 PM
3. Tuesday, July 22, 2025 11:56:05.801 PM, reboots back at Tuesday, July 22, 2025 11:56:45.926 PM
4.Wednesday, July 23, 2025 3:12:59.824 AM, reboots back at Wednesday, July 23, 2025 3:13:41.866 AM

I also made a plot of the jetson stats each time the reboot occurred and have shared those as well in the link. Also sharing the plots here for the first time the reboot happens. Other plots follow a similar trend every time the cameras reboot.

cpu_usage_per_core
cpu_usage_per_core700×500 80.7 KB

gpu_usage
gpu_usage700×500 35.9 KB

power_consumption
power_consumption700×500 34.5 KB

ram_usage
ram_usage700×500 21.4 KB

temperatures
temperatures700×500 39.7 KB

6

A question to create the streaming, do you run three instances of the streaming script or have you modified it to create 3 parallel threads in the same process?

7

Hi@Myzhar, we run the streaming_sender.py script for each of the cameras separately. So they are all different processes

8

I reported the issue to the ZED SDK team to try to reproduce and fix it.

Meanwhile, please upgrade to the ZED X Driver v1.3.1 and ZED SDK v5.0.4 to verify if the problem persists with the latest available versions.

9

Hi @Myzhar, the test results I posted above in the thread (HD1200@30fps test) was with driver v1.3.1

10

Hi @Myzhar, I ran some more tests with the upgraded SDK 5.0.5 and the link driver v1.3.1 and still observe the CAM REBOOTING error.

I also noticed that when streaming the cameras with the latest SDK and link driver (HD1200@60FPS), I came across segmentation fault issue which we saw when SDK5.0.0 was released and the same was reported here.

11

Hi @Myzhar
I have an adjacent question regarding the rebooting state of the camera, what would be the best thing to do when getting this error code?
When using Python, I tried to close the connection once this error was being raised during streaming and I then faced a Segmentation Fault.
Would just waiting be the correct handling of this error or what can I do in my software when facing this error? I could not find any information regarding this in the docs.
Thank you! :slight_smile:

12

Normally, you must wait for it to recover. 20-30 sec is the expected period.

1 Like
13

Hi @Myzhar , I wanted to check if there are any updates on this topic. Were you able to reproduce this issue on your end?

14

Hi @abhishekpavani,

@Myzhar is unfortunately not available at the moment but I’ll try my best to help out !

We’ve made some progress on the topic of long term capture stability of multiple GMSL cameras on a Jetson. The solution is being thoroughly tested and reviewed at the moment and we plan to release it as part of the next 5.1 release.

→ With that, you should be able to have your camera running 24/7 without issues.

Regarding the segfault you mentioned, do I understand it right that it happens at the moment you see the (Argus) Error Timeout and (Argus) Error InvalidState? If so, this will also be fixed by the solution mentioned above

15

Hi @LuckyJ ,

We’ve made some progress on the topic of long term capture stability of multiple GMSL cameras on a Jetson. The solution is being thoroughly tested and reviewed at the moment and we plan to release it as part of the next 5.1 release.

We will be looking forward for that release version.

Regarding the segfault you mentioned, do I understand it right that it happens at the moment you see the (Argus) Error Timeout and (Argus) Error InvalidState ? If so, this will also be fixed by the solution mentioned above

Yes the segfault happens after we see the (Argus) Error Timeout and (Argus) Error InvalidState.

But one thing to note here is that during the entire duration the cameras stream, there are many instances when we see the (Argus) Error Timeout and (Argus) Error InvalidState but the cameras do not segfault. However I would like to reiterate that the segfault has always happened after seeing one of the instances of the (Argus) Error Timeout and (Argus) Error InvalidState

16

Thank you for the precision! I confirm that this behavior will be fixed with the solution mentioned above