Neural Depth Model optimization error

Hello. I’ve recently started playing around with your new Neural depth mode. I’ve already tested it on Xavier AGX that we use for our Robotic AUV student project - It’s truly impressive! Amazing job! Today I wanted to make some dev on my personal workstation with RTX 3080 / Ryzen 5900 but I got hit by this traceback during optimization:

patryk@patryk-desktop:/usr/local/zed$ ./tools/ZED_Depth_Viewer 
Optimizing: NEURAL DEPTH...
 |[ZED][INFO] Please wait while the AI model is being optimized for your graphics card
 This operation will be run only once and may take a few minutes
|Stack trace (most recent call last) in thread 234631:
#31   Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in 
#30   Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7fafa66f6162, in clone
#29   Object "/lib/x86_64-linux-gnu/libpthread.so.0", at 0x7fafb3656608, in 
#28   Object "/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7fafa6bd39d1, in 
#27   Object "/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7fafa6bd6f81, in 
#26   Object "./tools/ZED_Depth_Viewer", at 0x448817, in 
#25   Object "./tools/ZED_Depth_Viewer", at 0x44998f, in 
#24   Object "/usr/local/zed/lib/libsl_zed.so", at 0x7fafa8a773e5, in ai::AILoader::optimAIModel(sl::AI_MODELS, int, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, int)
#23   Object "/usr/local/zed/lib/libsl_ai.so", at 0x7faf9ada2a0d, in 
#22   Object "/usr/local/zed/lib/libsl_ai.so", at 0x7faf9ada8eee, in 
#21   Object "/usr/local/zed/lib/libsl_ai.so", at 0x7faf9ad4cdde, in 
#20   Object "/usr/local/zed/lib/libsl_ai.so", at 0x7faf9ad4a545, in 
#19   Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf76a287f1, in 
#18   Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf76a23370, in 
#17   Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf7675f7db, in 
#16   Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf7675b08c, in 
#15   Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf768c22cd, in 
#14   Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf768c044a, in 
#13   Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf769c320a, in 
#12   Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf767614c3, in 
#11   Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf76734973, in 
#10   Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf769e7588, in 
#9    Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf769e6973, in 
#8    Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf767356e4, in 
#7    Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf769e78c3, in 
#6    Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf769fe8d9, in 
#5    Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf769fe6e5, in 
#4    Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf76abf5f1, in 
#3    Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf7674f30e, in 
#2    Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf762c84a3, in 
#1    Object "/usr/local/cuda-11.4/lib64/libnvinfer.so.8", at 0x7faf76671a1f, in 
#0    Object "/lib/x86_64-linux-gnu/libcuda.so.1", at 0x7fafb1dd40d2, in 
Bus error (Signal sent by the kernel [(nil)])
Bus error (core dumped)

Header from installation:

ZED_SDK_Ubuntu20_cuda11.5_v3.7.2.run 
Verifying archive integrity...  100%   MD5 checksums are OK. All good.
Uncompressing 'ZED camera SDK by Stereolabs'  100%  
Ubuntu version 20.04 detected. OK
To continue you have to accept the EULA. Accept  [Y/n] ?Y
Installing...
Installation path: /usr/local/zed
[sudo] password for patryk: 
Checking CUDA version...
OK: Found CUDA 11.4
Do you want to install the static version of the ZED SDK (AI module will still requires libsl_ai.so) [Y/n] ?Y
Do you want to install the Object Detection module (recommended), cuDNN 8.2 and TensorRT 8.2 will be installed [Y/n] ?Y

I’ve already installed cuDNN from libcudnn8_8.2.4.15-1+cuda11.4_amd64.deb. Any ideas how to resolve this issue?

It seems that there is a conflict with the CUDNN library version.
Can you please try to ptimize the NEURAL depth mode from the command line by running the command
ZED_Diagnostic -nrlo
and post here the output?

Thanks for your reply!
It finished successfully this time. Could you please tell me how to reset this optimization? I could see depth model in zed/resources, but removal of this file didn’t cause re-optimization

Hello, the models are indeed in zed/resources. There should be two files per model, one that is downloaded (.model) and one that is optimized (.model_optimized-xxxxxxxxx). It’s the one that is optimized that should be deleted.

1 Like

You can also use the command ZED_Diagnostic -aic to clean the models and their optimizations

1 Like

I encountered the same issue and ZED_Diagnostic -aic followed by ZED_Diagnostic -nrlo made it appear that the operation was successful:

[ZED_Diagnostic] Optimizing NEURAL depth mode for target device

Optimizing: NEURAL DEPTH...
 |[ZED][INFO] AI model not found, downloading
\[ZED][INFO] Done
[ZED][INFO] Done
[ZED][INFO] Please wait while the AI model is being optimized for your graphics card
 This operation will be run only once and may take a few minutes
 [ZED][INFO] Done

However when running in depth mode the depth image is just black in the zed_depth_viewer. And produces all nans when running with a python script so I’m not sure if the optimization was actually successful…

@moyodancer what ZED SDK version are you using?

3.7.2
The Zed diagnostic is telling me the neural model is optimized as well

@moyodancer please write an email to support@stereolabs.com and attach the report file generated by the ZED_Diagnostic tool.