Unable to complete Neural Model optimization for NEURAL DEPTH

Hi there,

I’m currently experiencing a problem while trying to enable neural mode in ZED_Depth_Viewer. The neural model optimization for neural depth fails to complete on ZED_Depth_Viewer GUI.

Executing ZED_Diagnostic --nrlo will result in a exit midway of the optimization process:

No error message are shown when the program exit. The program pretty much always exit when the progress bar is about 20 percent.

ZED_Diagnostic -aio optimizes other models fine until program progresses to optimizing REID ASSOCIATION when it seems to fail at 1.5%(without a ‘Done ! in 0min 43s’ like message),and then continues to optimize NEURAL DEPTH when it exits without error message at 20.9% .

ZED_Diagnostic GUI shows every model except for REID ASSOCIATION and NEURAL DEPTH optimized

System configuration:
Ubuntu 18.04
NVIDIA-SMI 450.172.01 Driver Version: 450.172.01
Cuda toolkit 10.2
Cudnn 8.5 (installed before SDK, ZED SDK seems to overwrite this to 8.2)

Has anyone experienced similar problem before?

Upgrade SDK version to v3.7.7 hasn’t solved this issue.

Hi @wericy
this seems to be a library issue.
Can you please start the ZED Diagnostic tool and send the report file that it generates to support@stereolabs.com citing this thread?

Sure, I have sent the report along with the email. Please let me know if you need any further information.

According to the Diagnostic Report that you sent, you are using the wrong configuration.

CUDA 10.2 does not support the Ampere architecture of the RTX GPU cards.
The RTX GPU models require CUDA 11, we suggest using the latest CUDA 11.7 with the latest SDK v3.7.7.

Thanks for the follow up!

I couldn’t find official sources indicating the incompatibility of CUDA 10.2 and Ampere cards.

According to documentation from NVIDIA:
CUDA 10.2 requires GPU driver version >= 440.33 which I satisfy.

If it is the CUDA related issue that’s the culprit here, I suspect it is because the use of cubin instead of PTX assembly during the compilation of ZED SDK for CUDA 10.2.

According to this doc:
“A CUDA application binary (with one or more GPU kernels) can contain the compiled GPU code in two forms, binary cubin objects and forward-compatible PTX assembly for each kernel.”
“A cubin generated for a certain compute capability is supported to run on any GPU with the same major revision and same or higher minor revision of compute capability.”
“Unlike cubin, PTX is forward-compatible. Meaning PTX is supported to run on any GPU with compute capability higher than the compute capability assumed for generation of that PTX. For example, PTX code generated for compute capability 7.x is supported to run on compute capability 7.x or any higher revision (major or minor), including compute capability 8.x.”

I would test CUDA 11 and report back here.

1 Like

The SDK team confirms that we don’t allow JIT since we don’t embed the PTX, so the RTX users must use CUDA >= 11.1

I appreciate your confirmation.

After upgrading to CUDA 11.7, NEURAL DEPTH model can be successfully optimized on my system.

Thanks again.

1 Like