Zedx argus socket failing after period of use and unable to recover

woudie-swap · September 11, 2024, 5:07am

Hi,

my team and I have been evaluating and developing with zedx cameras for the past few months, but we have been running into an issue where the argus socket seems to after a period of usage - which has varied from 40 minutes up to several hours depending on the system. Originally, we were testing/developing with 4 zedx cameras attached to a custom carrier board but are now testing with two cameras along with the drivers from stereolabs. I have attached a snippet of the logs we see when the crash occurs.

From the logs, we see something occurs causing camera grabs to fails and subsequently trigger a reboot but the reboot appears to trigger some kind of argus issue and then the application crashes (its a ROS1 zed wrapped node) until we restart everything.

current we are using the following:

nvidia jetson orin agx w/ jp5.1.2
zed sdk 4.1.3 and zedx driver 1.0.5 (for jp5.1.2)
our application is running inside a docker container

Any idea what might be going on here and what we could do to resolve this issue? we had this issue a long while ago but were unable to come to a solution until the recent zed4 and zedx driver updates (we were using zed sdk 4.0.2 with an older zedx driver (v0.5 or similar))
dual_zedx_crash.log (5.6 KB)

Myzhar · September 11, 2024, 7:20am

Hi @woudie-swap
I recommend upgrading to the latest Jetpack 6 and driver v1.1.0 for improved stability

woudie-swap · September 11, 2024, 2:55pm

Hi @Myzhar,

is that our best option - is this particularly for argus improvements in jetpack 6?

is there anything else we can do to at least ensure the camera can reboot without crashing our application?

Myzhar · September 12, 2024, 9:18am

Hi @woudie-swap
the ZED ROS Wrapper is no longer developed because of the close EOL of ROS 1.
We are only modifying it to make it compatible with the latest releases of the ZED SDK.

It is open source, so you can modify the error-handling part of the nodelet to better handle this behavior:

github.com

stereolabs/zed-ros-wrapper/blob/master/zed_nodelets/src/zed_nodelet/src/zed_wrapper_nodelet.cpp#L3996-L4081


      
          // ZED Grab
          mGrabStatus = mZed.grab(runParams);
          
          // cout << toString(grab_status) << endl;
          if (mGrabStatus != sl::ERROR_CODE::SUCCESS)
          {
            // Detect if a error occurred (for example: the zed have been disconnected) and re-initialize the ZED
          
            NODELET_INFO_STREAM_THROTTLE(1.0, "Camera grab error: " << sl::toString(mGrabStatus).c_str());
          
            std::this_thread::sleep_for(std::chrono::milliseconds(1));
            if (mGrabStatus != sl::ERROR_CODE::CAMERA_REBOOTING)
            {
              std::this_thread::sleep_for(std::chrono::milliseconds(50));
              continue;
            }
            else if (mSvoMode && mGrabStatus == sl::ERROR_CODE::END_OF_SVOFILE_REACHED)
            {
              NODELET_WARN("SVO reached the end. The node will be stopped.");
              mZed.close();

This file has been truncated. show original

ROS 1 is close to its end of life. In May 2025, Open Robotics will stop maintaining it and no more updates and fixes will be released.
My advice is to consider migrating to JP6 and to ROS 2 Humble which is natively supported by Ubuntu 22.04 on L4T36.3.