ZedX Camera driver failing to install due to copy of rootfs on SSD (Jetson)

anath93 · October 6, 2023, 11:20am

Getting error while installing ZedX camera driver on Jetson AGX Orin 64GB,

####################################
############ DMESG LOG #############
####################################

####################################
############ Driver list ###########
####################################

/usr/lib/modules/5.10.104-tegra/kernel/drivers/stereolabs/max96712/max96712.ko==> exists
/usr/lib/modules/5.10.104-tegra/kernel/drivers/stereolabs/zedx/sl_zedx.ko ==> exists

################################################
############ ZEDX Daemon Status LS #############
################################################

● zed_x_daemon.service - ZED-X Daemon service
Loaded: loaded (/etc/systemd/system/zed_x_daemon.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2023-04-25 21:37:46 PDT; 6min ago
Main PID: 1122 (ZEDX_Daemon)
Tasks: 3 (limit: 36321)
Memory: 10.7M
CGroup: /system.slice/zed_x_daemon.service
└─1122 /usr/local/zed/bin/ZEDX_Daemon

Apr 25 21:37:46 jetsonorin systemd[1]: Started ZED-X Daemon service.
Apr 25 21:37:46 jetsonorin ZEDX_Daemon[1122]: [ “Tue Apr 25 21:37:46 2023” ] ** Start ZED-X Daemon
Apr 25 21:37:47 jetsonorin ZEDX_Daemon[1122]: [ “Tue Apr 25 21:37:47 2023” ] Process “insmod /usr/lib/modules/5.10.104-tegra/kernel/drivers/stereolabs/max96712/max96712.ko” outputs “”
Apr 25 21:37:47 jetsonorin ZEDX_Daemon[1122]: [ “Tue Apr 25 21:37:47 2023” ] Process “rmmod sl_zedx” outputs “rmmod: ERROR: Module sl_zedx is not currently loaded\n”
Apr 25 21:37:47 jetsonorin ZEDX_Daemon[1122]: [ “Tue Apr 25 21:37:47 2023” ] Process “insmod /usr/lib/modules/5.10.104-tegra/kernel/drivers/stereolabs/zedx/sl_zedx.ko” outputs “”
Apr 25 21:37:47 jetsonorin ZEDX_Daemon[1122]: ** ZED-X Driver loaded
Apr 25 21:37:47 jetsonorin ZEDX_Daemon[1122]: [ “Tue Apr 25 21:37:47 2023” ] ** Created Endpoint “tcp://127.0.0.1:20026”

################################################
############ I2C Tracz /dev/i2C-30 #############
################################################

Error: Could not open file /dev/i2c-30' or /dev/i2c/30’: No such file or directory

################################################
############ I2C Tracz /dev/i2C-31 #############
################################################

Error: Could not open file /dev/i2c-31' or /dev/i2c/31’: No such file or directory

################################################
############ I2C Tracz /dev/i2C-32 #############
################################################

Error: Could not open file /dev/i2c-32' or /dev/i2c/32’: No such file or directory

################################################
############ I2C Tracz /dev/i2C-33 #############
################################################

Error: Could not open file /dev/i2c-33' or /dev/i2c/33’: No such file or directory
#########################

Myzhar · October 6, 2023, 12:13pm

Hi @anath93
please run the command $ sudo ZED_Calibration --dmesg and send the new log file.
sudo is required to get the full set of information

anath93 · October 6, 2023, 12:29pm

Please see attached.
dmesg.log (5.8 KB)

obraun-sl · October 6, 2023, 1:24pm

Hi,

Looking at the dmesg.log, it is not the correct dtb that is loaded when the Jetson is booting (it is still the original NVIDIA dtb).
When the driver is installed, it will modify the extlinux.conf to add the Stereolabs Label entry. We support having the rootfs on the SSD only when it is done by the NVIDIA SDK Manager (flashing on nvme instead of mmcblk).

If you have move the rootfs after flashing, then you need to update manually the extlinux.conf that will be effectively loaded (maybe in /media/xxxx/boot/extlinux/exlinux.conf) and add the FDT Entry :
FDT /boot/stereolabs/tegra234-p3701-0000-p3737-0000.dtb
instead of FDT /boot/dtb/kernel_tegra234-p3701-0000-p3737-0000.dtb

But currently the exlinux.conf is not coherent with your tree/architecture , so my advice is to reflash with SDK manager with the nvme as root target. When done, you will see that the extlinux.conf will have the
APPEND ${cbootargs} root=PARTUUID=<uuid>, .... which is indeed correct (the rootfs is on the part uuid of the nvme)

Let me know if that helps

anath93 · October 6, 2023, 1:58pm

I tried appending the line to exlinux.conf file, but no luck still. Re-Flash would be my last option to do at this point. Any other suggestions maybe before going that route ?

obraun-sl · October 6, 2023, 2:03pm

which extlinux.conf did you update? Can you post it here ?

anath93 · October 6, 2023, 2:07pm

I went in boot manager setup, on boot

obraun-sl · October 6, 2023, 2:11pm

Once booted, what is the result of
sudo dmesg | grep dts ?

anath93 · October 6, 2023, 2:16pm

obraun-sl · October 6, 2023, 2:44pm

From your dmesg image, it shows that the DTB loaded has ben build from
“/dvs/git/dirty/git_master_linux/…”
which is the original NVIDIA dtb.

so the command in boot manager setup did not work.

It will be more easy to reflash.

anath93 · October 10, 2023, 6:29pm

Re-flashing, with changing manually the extlinux file worked. But there is another issue, when ZED_Explorer is opened, and switch the mode to SVGA the camera crashes until next reboot.

Myzhar · October 11, 2023, 7:27am

Can you post the new dmesg.log file generated by sudo ZED_Diagnostic --dmesg after the flashing?

anath93 · October 11, 2023, 1:47pm

Please see attached.
dmesg.log (11.3 KB)

Myzhar · October 11, 2023, 2:29pm

The report shows that the DTS is still not correctly loaded:

[    0.004695] DTS File Name: /builds/sl/jetson-gmsl-kernel/src/kernel/kernel-5.10/arch/arm64/boot/dts/../../../../../../hardware/nvidia/platform/t23x/concord/kernel-dts/tegra234-p3701-0000-p3737-0000.dts

Is ZED_Explorer correctly displaying the camera stream before crashing?
Does ZED_Depth_Viewer work?

anath93 · October 11, 2023, 2:30pm

Yes they both work, but as soon as I switch modes it fails.

Something I would like to add:

I am on the latest driver with 5.1.2 jetpack. (SVGA @ 30 FPS), if I restart it works. I feel when the connection ends, its not closing the socket which is why this could happen ?

Also looks like the rgb/image_rect_color and camera_info and not synced too.

obraun-sl · October 12, 2023, 7:58am

You don’t really need to reboot.
Just doing $ sudo service nvargus-daemon restart will unlock if the closing is not properly done.

anath93 · October 12, 2023, 12:08pm

Thank you for that, this also happens when launching as a node too and also there is sync issues between rgb image and camera calibration topics as I get error back from my ros2 node, do you want me open bug ticket on GITHUB for this ?

obraun-sl · October 12, 2023, 12:31pm

Just tested with a ZEDBox (Orin NX) and a ZED-X with the latest driver (0.5.6) and latest ZED SDK (4.0.7) and I did not encounter such issues when switching resolution in ZED Explorer or Depth Viewer
(Make sure that you are not killing the software at the end, otherwise the close function won’t be called and you will need to restart the nvargus daemon).

If you have questions/bugs related to ROS specifically, yes it is better to open an issue on the ROS/ROS2 github repository

Myzhar · October 12, 2023, 12:56pm

Can you explain better this? How did you check this “sync issue”?
Images and Camera Info messages are published by the node in the same moment by leveraging the image_transport package. It is practically impossible that they are not synchronized.

Please create a new forum thread to discuss this problem

anath93 · October 12, 2023, 1:09pm

@Myzhar That’s what I was thinking but I will open another forum page to give more insights on that.