Zed with PyTorch/Mask R-CNN

Software Python
, ,
1

I have ran through this tutorial (How to Use PyTorch with ZED | Stereolabs) and completed all the prior steps, but when I get to the Mask R-CNN Install step, I get an error:

maskrcnn_benchmark\csrc\cuda\ROIAlign_cuda.cu(5): fatal error C1083: Cannot open include file: 'THC/THC.h': No such file or directory

I am using cuda 12.1, should I downgrade pytorch and cuda?

2

I ran into the same issue when trying to use pytorch after installing it via pip.
You could try compiling pytorch ā€˜manuallyā€™ with the latest version of cuda.

conda install:

conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch-nightly -c nvidia

pip:

pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121

source: Start Locally | PyTorch

3

I already tried that and still got the error.

However, I downgraded to cuda 11.7 and seems to get passed that error, just to hit another:

Traceback (most recent call last):
  File "C:\Users\joini\mask\zed-pytorch-master\zed_object_detection.py", line 6, in <module>
    from predictor import COCODemo
  File "C:\Users\joini\mask\zed-pytorch-master\predictor.py", line 6, in <module>
    from maskrcnn_benchmark.modeling.detector import build_detection_model
ModuleNotFoundError: No module named 'maskrcnn_benchmark'

And when I try adding the following line of code to the predictor.py:

sys.path.append("maskrcnn_benchmark/")

I get the following error:

ImportError: cannot import name '_C' from 'maskrcnn_benchmark'

Do you know why this is happening?

4

From this link, it seems like THC.h is not in pytorch anymore.
Mask R-CNN seems also deprecated. Maybe you should try its successor ?

5

I am just learning so was somewhat relying/leaning on that tutorial. Does much change to use detectron2?

6

I donā€™t know, Iā€™ve never used it. What are you trying to achieve ? Are you trying to train your own object detection models ?

7

No, Iā€™m trying to get a pre-trained model ingested into the custom detector in Unity to detect water bottles, etc. for a mixed reality ocean robotic cleanup project. Donā€™t need to train them myself as most are trained on bottles e.g. yolo. But I need segmentation masks as well, which yolo doesnā€™t support.

Any pointers?

Is there some way to use the yolo custom detector examples and obtain masks of the bottles somehow using the ZED SDK?

Edit: I see YOLO v8 supports instance seg. so is this the route I should explore?

8

So I try with the yolo v8 tutorial and get the following error:

(zed) C:\Program Files (x86)\ZED SDK\samples\object detection\custom detector\python\pytorch_yolov8>python detector.py --weights yolov8m.pt # [ā€“img_size 512 --conf_thres 0.1 --svo path/to/file.svo]
Traceback (most recent call last):
File ā€œC:\Program Files (x86)\ZED SDK\samples\object detection\custom detector\python\pytorch_yolov8\detector.pyā€, line 9, in
import pyzed.sl as sl
ImportError: DLL load failed while importing sl: The specified procedure could not be found.

9

Iā€™m not a Yolo expert, canā€™t really tell. However you donā€™t need Pytorch to run our Custom Object detector tutorial.

It seems you also have an issue with your python wrapper (the import error you have). Can you install it again with get_python_api.py ? Are you using conda ?

10

Yes Iā€™m using conda, and also use the get_python_api.py. Will try again with fresh conda env.

12

Thanks for your help thus far; I got it installed. However, when trying to run the detector using:

python detector.py --weights yolov8m.pt

it throws the following error:

Traceback (most recent call last):
  File "C:\Program Files (x86)\ZED SDK\samples\object detection\custom detector\python\pytorch_yolov8\detector.py", line 9, in <module>
    import pyzed.sl as sl
ImportError: DLL load failed while importing sl: The specified procedure could not be found.

But pyzed is installed.

13

I have followed the steps in the solution identified in these posts but it has not resolved it for me.

14

Does it work outside of conda ?
Did you install the wrapper with get_python_api or build it from source ?

15

Nope, throws the same error outside of conda.
I installed using the wrapper. When I try to build from source I get lots of cyphon compile errorsā€“too many to paste here, but this is the end of them:

------------------------------------------------------------

pyzed\sl.pyx:10455:89: 'c_REFERENCE_FRAME' is not a type identifier

Error compiling Cython file:
------------------------------------------------------------
...
    # Get the Fused Position of the camera system
    # \param camera_pose will contain the camera pose in world position (world position is given by the calibration of the cameras system)
    # \param reference_frame defines the reference from which you want the pose to be expressed. Default : \ref REFERENCE_FRAME "REFERENCE_FRAME::WORLD".
    # \return POSITIONAL_TRACKING_STATE is the current state of the tracking process
    def get_position(self, camera_pose : Pose, reference_frame : REFERENCE_FRAME = REFERENCE_FRAME.WORLD, uuid: CameraIdentifier = CameraIdentifier(), position_type : POSITION_TYPE = POSITION_TYPE.FUSION):
        return POSITIONAL_TRACKING_STATE(<int>self.fusion.getPosition(camera_pose.pose, <c_REFERENCE_FRAME>(<int>reference_frame.value), uuid.cameraIdentifier, <c_POSITION_TYPE>(<int>position_type.value)))

                                         ^
------------------------------------------------------------

pyzed\sl.pyx:10455:161: 'c_POSITION_TYPE' is not a type identifier
Traceback (most recent call last):
  File "C:\Users\xxx\xxx\Documents\code\xxx\ZED\zed-python-api-master\src\setup.py", line 194, in <module>
    extList = cythonize(extension, compiler_directives=cython_directives)
  File "C:\Users\joini\AppData\Local\Programs\Python\Python310\lib\site-packages\Cython\Build\Dependencies.py", line 1134, in cythonize
    cythonize_one(*args)
  File "C:\Users\joini\AppData\Local\Programs\Python\Python310\lib\site-packages\Cython\Build\Dependencies.py", line 1301, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: pyzed/sl.pyx
16

I guess the sources are not up to date. get_python_api is the safest way to go, anyway.

To debug your python module, can we go through a few steps ?

  • Make sure you use the latest SDK (4.0.7)
  • Run ZED Depth Viewer. If it works, the SDK works.
  • in the same conda env, or outside, run python get_python_zpi.py and then python <your python script. Donā€™t leave the terminal, just to be sure.
1 Like
17

OK, looks like some progress has been made :slight_smile:

It attempts to run Ultralytics now, which is good news, but now throws a new error:

ImportError: cannot import name '_c_internal_utils' from partially initialized module 'matplotlib' (most likely due to a circular import) (C:\Users\joini\anaconda3\envs\zedyolo\lib\site-packages\matplotlib\__init__.py)
18

Itā€™s probably because you have the wrong version of something - can be matplotlib, yolo, etc.

19

For some reason there was a series of modules (matplotlib, opencv, pillow) that needed to be uninstalled and re-installed.

Things are looking good thanks! When I run:

python detector.py --weights yolov8m.pt

The following happens in the terminal:

Intializing Network...
Initializing Camera...
Downloading https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8m.pt to 'yolov8m.pt'...
Initialized Camera

and the ZED object detection window opens. However, the window is a black screen and eventually becomes unresponsive.

20

Hi, since this has become a new issue, I have set this topic as solved and started a new thread for the issue with the yolo detector.

If you can provide any insight into that, Iā€™d appreciate the help, as since it is not throwing any errors, I have nothing to go on.

Thanks again for your help with the python wrapper etc. :slight_smile:

21

Seen that. Weā€™ll continue from there.