Nvargus camera failure after some time

Software
,
1

Hi everyone,

I am posting here hoping to find clarification regarding an issue I am experiencing since a few months, quite annoying and up to now without a traceable cause. It happens at random but when it does it is quite destructive.

PROBLEM DESCRIPTION
Camera is working fine with positional tracking, depth, sensors being sampled correctly by a software I wrote accoring to your documentation and sl/Camera.hpp; after some time it randomly fails with these symptoms:

  • grab() returns ERROR CAMERA_REBOOTING;
  • the nvargus error messages reported below appear in the console log;
  • camera has to be restarted to become usable again, i.e. my software has to be forcibly terminated then re-started to re-open the camera which then works fine again.

NVARGUS ERROR MESSAGES
Just an excerpt that I was able to extract.
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
(Argus) Error FileOperationFailed: Failed socket read: Connection reset by peer (in src/rpc/socket/common/SocketUtils.cpp, function readSocket(), line 79)
(Argus) Error FileOperationFailed: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 277)
(Argus) Error FileOperationFailed: Receive worker failure, notifying 2 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 350)
(Argus) Error InvalidState: Argus client is exiting with 2 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 366)
(Argus) Error FileOperationFailed: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 379)
(Argus) Error FileOperationFailed: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error FileOperationFailed: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)

SETUP DESCRIPTION

  • Nvidia Jetson Orin NX 16GB (custom carrier)
  • ZED Link Mono capture card connected to CSI port on Orin, 9 V power supplied from battery
  • JetPack 6.2.1 (full installation by Nvidia SDKManager)
  • ZED X/X Mini camera (tested with both, happens with both)
  • ZED Link driver 1.3.1 (correctly installed from .deb package)
  • ZED SDK 5.0.3
  • Software written in C++ using ZED SDK, running inside a Docker container set up following your documentation (full privileges, udev rules installed on the host, systemd service volume mounted etc.)
  • Running positional tracking, resolution HD1200, NEURAL_LIGHT depth mode, IMU sampling

Many thanks to everyone who is willing to help,

RM

2

Hi @robmasocco
Welcome to the Sterelabs community.

The described issue seems to be caused by an unstable GMSL2 connection.
I recommend you install the latest ZED SDK v5.0.7 for reliable handling of these types of conditions.

Who is the manufacturer? Are you using a custom driver?

3

Hi @Myzhar , thank you for replying.

The carrier is a Holybro Jetson Baseboard, which is essentially an Orin NX with a custom carrier having a Pixhawk board wired on. It can be flashed as a normal Orin NX using Nvidia SDKManager.
I am using a ZED Link Mono capture card connected to a CSI port and receiving an external power supply at 9 V from a battery, through the carrier’s included step-down. The camera is connected to the Mono card using a 1-1 cable we bought from you (we bought all your hardware from your web store).
I have installed the ZED Link driver 1.3.1 from your .deb packages (not those for the ZED Boxes but the generic ones).

Additionally, I have seen this problem occur with the same randomness also on another platform where we have a ZED X connected to a ZED Link Duo capture card, connected and securely attached to a Jetson Orin AGX devkit (official one bought from Nvidia).

4

This board is not supported by our drivers. It requires a custom driver to work, but it’s not available.