Issue with neural network using detectron2

jlee · September 23, 2022, 5:29pm

Hi,

I am testing two famous machine learning packages with ZED2i, such as detectron2 and yolov7.
yolov7 model works pretty well without any issue. But I had an issue when I use detectron2.

I trained my custom model by detectron2 and tried to predict objects based on the ZED camera image using OpenCV. Simultaneously, I tried to display a 3D point cloud in the OpenGL viewer. Unfortunately, I faced the error below.

RuntimeError: CUDA error: invalid argument
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

This error was not shown when I passed my model through CPU. So, this error only occur when both ZED and Detectron use CUDA.

I was wondering if anyone succeed to integrate Detectron2 with ZED SDK.
Should I refer zed-pytorch (GitHub - stereolabs/zed-pytorch: 3D Object detection using the ZED and Pytorch)?
Any comments would be appreciated.

Thank you.

Best,
Jay

Myzhar · September 23, 2022, 5:58pm

Hi @jlee
I think that this Github issue can contain useful information regarding your problem:

github.com/stereolabs/zed-python-api

cudnn error when running zed api alongside a pytorch model

opened 09:34PM - 27 Sep 20 UTC

closed 01:03AM - 27 Apr 22 UTC

ffletcherr

closed_for_stale Stale

Hi, I use ZED2 camera and zed api in python and have a pytorch model running i…n the same script. But I get the following cuDNN error. However, when running them seperately, at the same time in different scripts, no error pops up! (Both on an nvidia 2070 super GPU) I need to run both of the codes in the same script since it's a sequence (I get the images and disparities from the zed api and then run some deep learning models on them). ``` Traceback (most recent call last): File "main.py", line 92, in <module> yoloOutput = detector.detect(left) File "E:\load\final\yolo.py", line 41, in detect pred = self.yolo_model(img, augment=False)[0] File "C:\Users\user\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "E:\load\final\models\yolo.py", line 113, in forward return self.forward_once(x, profile) # single-scale inference, train File "E:\load\final\models\yolo.py", line 133, in forward_once x = m(x) # run File "C:\Users\user\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "E:\load\final\models\common.py", line 32, in fuseforward return self.act(self.conv(x)) File "C:\Users\user\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "C:\Users\user\anaconda3\lib\site-packages\torch\nn\modules\conv.py", line 419, in forward return self._conv_forward(input, self.weight) File "C:\Users\user\anaconda3\lib\site-packages\torch\nn\modules\conv.py", line 415, in _conv_forward return F.conv2d(input, weight, self.bias, self.stride, RuntimeError: Unable to find a valid cuDNN algorithm to run convolution ``` CUDA version : 10.2 cudnn version : 8.0.3.33 pytorch version: 1.6.0 ZED SDK version: 3.2.2

momi193 · February 13, 2023, 1:03pm

@jlee I am using the yolov5 sample and know of the custom detection documentation: Using the Object Detection API with a Custom Detector | Stereolabs

How did you get yoloV7 and therefore maybe yoloV8 running? I am still working my way through the example code and would also like to better understand the relation between the yolo object info and the SDK objects[] array. I would like to log the distance in vector space and the yolo object class.

If you are able to help me/ point me in a good direction to start, I would really appreciate that

Myzhar · February 13, 2023, 1:38pm

Have you tried to adapt the GitHub repository code to the new inference engine?

momi193 · February 13, 2023, 2:14pm

I am in the process of trying that but if someone has successfully done that, it would safe a lot of time as I am not a native computational engineer.

CarloGem · March 16, 2023, 10:16am

Hi! Did you manage to run the yolov7 model inference on the Zed camera?
I’m facing the same issue on a Zed2 camera and I don’t really understand how to adapt the existing example of object detection (using yolov5) to yolov7.
If anyone could point me to the right direction I would greatly appreciate it.