Zed + TensorRT problems: invalidating cuda context handle

Software Python
,
1

Does the Zed SDK activate or clear existing tensor_rt contexts?

(following up on our question last year)

Situation:

  • Running Zed + cuda accelerated YoloV4 works fine
  • Loading saved video frames + tensor_rt accelerated YoloV4 works fine
  • But Zed + tensor_rt accelerated YoloV4 runs into a problem with an invalidated cuda context

System Info

  • pyzed: 3.7
  • Hardware: Jetson Xavier
  • TensorRT version: 8.0
  • Cuda: cuda_10.2_r440.TC440_70.29663091_0
  • python: 3.6.9

Notes:

  • Error text:
    [TensorRT] ERROR: 1: [reformat.cu::NCHWToNHWC::1405] Error Code 1: Cuda Runtime (invalid resource handle)
  • From other online sources, it seems this error typically happens when two systems are trying to use TensorRT, like using pycuda and torch indenpendently. However, we only have YoloV4 using TensorRT
  • The TensorRT warning (in the image below) should be a false positive. We never transferred a .engine cache, and the warning is known to be faulty.
  • Attempting to create a cuda context after the existing handle is broken, creates a different error: [hardwareContext.cpp::configure::92] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)

Screen Shot 2022-05-01 at 3.43.39 PM
Screen Shot 2022-05-01 at 3.43.39 PM2396×1016 407 KB

2

If you’re not using NEURAL depth nor any models of the Object detection module, there’s no TensorRT in use in the ZED SDK.

Are you sure this is not a CUDA context issue or maybe a stream issue?

How did you manage the CUDA context? You can either create it, then give it to the ZED SDK (InitParameters.sdk_cuda_ctx) or let the ZED SDK creates one then you should not call any cudaSetDevice or equivalent, except if you’re using multiple threads.

3

I doubt its a stream issue, realsense2 + YoloV4 tensorRT works, and Zed works with tensorRT off

Passing in the cuda context sounds promising, but I get “not found” for all the ways I’ve attempted to use the API

  • init_params = sl.InitParameters(sdk_verbose=True)
    init_params.sdk_cuda_ctx = our_cuda_ctx
  • init_params = sl.InitParameters(sdk_verbose=True)
    init_params.sdk_cuda_ctx_ = our_cuda_ctx
  • init_params = sl.InitParameters(sdk_verbose=True, sdk_cuda_ctx=our_cuda_ctx)
  • init_params = sl.InitParameters(sdk_verbose=True, sdk_cuda_ctx_=cuda_ctx)

e.g.

AttributeError: 'pyzed.sl.InitParameters' object has no attribute 'sdk_cuda_ctx'
# or 
TypeError: __cinit__() got an unexpected keyword argument 'sdk_cuda_ctx_'

Is there only a C++ interface for passing cuda?

4

Ah ok sorry I thought you were using the C++ API, in python, it currently doesn’t exist, you need a workaround such as this GitHub - frk2/cuda-fun: Cython cuda wrapper to switch contexts for running multiple contexts app in the same process