# Zed with PyTorch/Mask R-CNN

**URL:** <https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581>\
**Category:** Python\
**Tags:** python, sdk, windows\
**Created:** [September 26, 2023, 10:31pm UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581 "2023-09-26T22:31:29Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![a11s](https://community.stereolabs.com/letter_avatar_proxy/v4/letter/a/df705f/32.png) [@a11s](https://community.stereolabs.com/u/a11s)\
**Post date:** [September 26, 2023, 10:31pm UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/1 "2023-09-26T22:31:29Z")

</div>

I have ran through this [tutorial](https://www.stereolabs.com/docs/pytorch/) ([How to Use PyTorch with ZED | Stereolabs](https://www.stereolabs.com/docs/pytorch/)) and completed all the prior steps, but when I get to the [Mask R-CNN Install](https://www.stereolabs.com/docs/pytorch/#mask-r-cnn-installation) step, I get an error:

```auto
maskrcnn_benchmark\csrc\cuda\ROIAlign_cuda.cu(5): fatal error C1083: Cannot open include file: 'THC/THC.h': No such file or directory

```

I am using cuda 12.1, should I downgrade pytorch and cuda?

---

<div class="post-metadata">

**Author:** ![fdunbar](https://community.stereolabs.com/letter_avatar_proxy/v4/letter/f/b782af/32.png) [@fdunbar](https://community.stereolabs.com/u/fdunbar)\
**Post date:** [September 26, 2023, 11:22pm UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/2 "2023-09-26T23:22:47Z")

</div>

I ran into the same issue when trying to use pytorch after installing it via pip.  
You could try compiling pytorch ‘manually’ with the latest version of cuda.

conda install:

```auto
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch-nightly -c nvidia

```

pip:

```auto
pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121

```

source: [Start Locally | PyTorch](https://pytorch.org/get-started/locally/)

---

<div class="post-metadata">

**Author:** ![a11s](https://community.stereolabs.com/letter_avatar_proxy/v4/letter/a/df705f/32.png) [@a11s](https://community.stereolabs.com/u/a11s)\
**Post date:** [September 27, 2023, 12:48am UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/3 "2023-09-27T00:48:22Z")

</div>

I already tried that and still got the error.

However, I downgraded to cuda 11.7 and seems to get passed that error, just to hit another:

```auto
Traceback (most recent call last):
  File "C:\Users\joini\mask\zed-pytorch-master\zed_object_detection.py", line 6, in <module>
    from predictor import COCODemo
  File "C:\Users\joini\mask\zed-pytorch-master\predictor.py", line 6, in <module>
    from maskrcnn_benchmark.modeling.detector import build_detection_model
ModuleNotFoundError: No module named 'maskrcnn_benchmark'

```

And when I try adding the following line of code to the predictor.py:

```auto
sys.path.append("maskrcnn_benchmark/") 

```

I get the following error:

```auto
ImportError: cannot import name '_C' from 'maskrcnn_benchmark'

```

Do you know why this is happening?

---

<div class="post-metadata">

**Author:** ![alassagne](https://community.stereolabs.com/user_avatar/community.stereolabs.com/alassagne/32/2428_2.png) [@alassagne](https://community.stereolabs.com/u/alassagne)\
**Post date:** [September 27, 2023, 9:01am UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/4 "2023-09-27T09:01:41Z")

</div>

> [@a11s](#):
>
> `Cannot open include file: 'THC/THC.h'`

From [this link](https://github.com/open-mmlab/mmdetection3d/issues/1332#issuecomment-1085991179), it seems like THC.h is not in pytorch anymore.  
[Mask R-CNN](https://github.com/facebookresearch/maskrcnn-benchmark) seems also deprecated. Maybe you should try its [successor](https://github.com/facebookresearch/detectron2) ?

---

<div class="post-metadata">

**Author:** ![a11s](https://community.stereolabs.com/letter_avatar_proxy/v4/letter/a/df705f/32.png) [@a11s](https://community.stereolabs.com/u/a11s)\
**Post date:** [September 27, 2023, 11:46am UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/5 "2023-09-27T11:46:18Z")

</div>

I am just learning so was somewhat relying/leaning on that tutorial. Does much change to use detectron2?

---

<div class="post-metadata">

**Author:** ![alassagne](https://community.stereolabs.com/user_avatar/community.stereolabs.com/alassagne/32/2428_2.png) [@alassagne](https://community.stereolabs.com/u/alassagne)\
**Post date:** [September 27, 2023, 4:03pm UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/6 "2023-09-27T16:03:41Z")

</div>

I don’t know, I’ve never used it. What are you trying to achieve ? Are you trying to train your own object detection models ?

---

<div class="post-metadata">

**Author:** ![a11s](https://community.stereolabs.com/letter_avatar_proxy/v4/letter/a/df705f/32.png) [@a11s](https://community.stereolabs.com/u/a11s)\
**Post date:** [September 27, 2023, 4:29pm UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/7 "2023-09-27T16:29:45Z")

</div>

No, I’m trying to get a pre-trained model ingested into the custom detector in Unity to detect water bottles, etc. for a mixed reality ocean robotic cleanup project. Don’t need to train them myself as most are trained on bottles e.g. yolo. But I need segmentation masks as well, which yolo doesn’t support.

Any pointers?

Is there some way to use the yolo custom detector examples and obtain masks of the bottles somehow using the ZED SDK?

Edit: I see YOLO v8 supports instance seg. so is this the route I should explore?

---

<div class="post-metadata">

**Author:** ![a11s](https://community.stereolabs.com/letter_avatar_proxy/v4/letter/a/df705f/32.png) [@a11s](https://community.stereolabs.com/u/a11s)\
**Post date:** [September 27, 2023, 8:46pm UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/8 "2023-09-27T20:46:33Z")

</div>

So I try with the yolo v8 tutorial and get the following error:

(zed) C:\Program Files (x86)\ZED SDK\samples\object detection\custom detector\python\pytorch\_yolov8\>python detector.py --weights yolov8m.pt # [–img\_size 512 --conf\_thres 0.1 --svo path/to/file.svo]  
Traceback (most recent call last):  
File “C:\Program Files (x86)\ZED SDK\samples\object detection\custom detector\python\pytorch\_yolov8\detector.py”, line 9, in   
import pyzed.sl as sl  
ImportError: DLL load failed while importing sl: The specified procedure could not be found.

---

<div class="post-metadata">

**Author:** ![alassagne](https://community.stereolabs.com/user_avatar/community.stereolabs.com/alassagne/32/2428_2.png) [@alassagne](https://community.stereolabs.com/u/alassagne)\
**Post date:** [September 28, 2023, 11:28am UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/9 "2023-09-28T11:28:21Z")

</div>

I’m not a Yolo expert, can’t really tell. However you don’t need Pytorch to run our Custom Object detector tutorial.

It seems you also have an issue with your python wrapper (the import error you have). Can you install it again with `get_python_api.py` ? Are you using conda ?

---

<div class="post-metadata">

**Author:** ![a11s](https://community.stereolabs.com/letter_avatar_proxy/v4/letter/a/df705f/32.png) [@a11s](https://community.stereolabs.com/u/a11s)\
**Post date:** [September 28, 2023, 1:08pm UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/10 "2023-09-28T13:08:55Z")

</div>

Yes I’m using conda, and also use the get\_python\_api.py. Will try again with fresh conda env.

---

<div class="post-metadata">

**Author:** ![a11s](https://community.stereolabs.com/letter_avatar_proxy/v4/letter/a/df705f/32.png) [@a11s](https://community.stereolabs.com/u/a11s)\
**Post date:** [September 28, 2023, 10:25pm UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/12 "2023-09-28T22:25:37Z")

</div>

Thanks for your help thus far; I got it installed. However, when trying to run the detector using:

```auto
python detector.py --weights yolov8m.pt

```

it throws the following error:

```auto
Traceback (most recent call last):
  File "C:\Program Files (x86)\ZED SDK\samples\object detection\custom detector\python\pytorch_yolov8\detector.py", line 9, in <module>
    import pyzed.sl as sl
ImportError: DLL load failed while importing sl: The specified procedure could not be found.

```

But pyzed is installed.

---

<div class="post-metadata">

**Author:** ![a11s](https://community.stereolabs.com/letter_avatar_proxy/v4/letter/a/df705f/32.png) [@a11s](https://community.stereolabs.com/u/a11s)\
**Post date:** [September 29, 2023, 3:32am UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/13 "2023-09-29T03:32:03Z")

</div>

I have followed the steps in the solution identified in [these posts](https://community.stereolabs.com/t/still-the-classic-problem-pyzed-sl-importerror-who-can-help/1016/2) but it has not resolved it for me.

---

<div class="post-metadata">

**Author:** ![alassagne](https://community.stereolabs.com/user_avatar/community.stereolabs.com/alassagne/32/2428_2.png) [@alassagne](https://community.stereolabs.com/u/alassagne)\
**Post date:** [September 29, 2023, 7:10am UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/14 "2023-09-29T07:10:50Z")

</div>

Does it work outside of conda ?  
Did you install the wrapper with get\_python\_api or build it from source ?

---

<div class="post-metadata">

**Author:** ![a11s](https://community.stereolabs.com/letter_avatar_proxy/v4/letter/a/df705f/32.png) [@a11s](https://community.stereolabs.com/u/a11s)\
**Post date:** [September 29, 2023, 2:19pm UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/15 "2023-09-29T14:19:57Z")

</div>

Nope, throws the same error outside of conda.  
I installed using the wrapper. When I try to build from source I get lots of cyphon compile errors–too many to paste here, but this is the end of them:

```auto
------------------------------------------------------------

pyzed\sl.pyx:10455:89: 'c_REFERENCE_FRAME' is not a type identifier

Error compiling Cython file:
------------------------------------------------------------
...
    # Get the Fused Position of the camera system
    # \param camera_pose will contain the camera pose in world position (world position is given by the calibration of the cameras system)
    # \param reference_frame defines the reference from which you want the pose to be expressed. Default : \ref REFERENCE_FRAME "REFERENCE_FRAME::WORLD".
    # \return POSITIONAL_TRACKING_STATE is the current state of the tracking process
    def get_position(self, camera_pose : Pose, reference_frame : REFERENCE_FRAME = REFERENCE_FRAME.WORLD, uuid: CameraIdentifier = CameraIdentifier(), position_type : POSITION_TYPE = POSITION_TYPE.FUSION):
        return POSITIONAL_TRACKING_STATE(<int>self.fusion.getPosition(camera_pose.pose, <c_REFERENCE_FRAME>(<int>reference_frame.value), uuid.cameraIdentifier, <c_POSITION_TYPE>(<int>position_type.value)))

                                         ^
------------------------------------------------------------

pyzed\sl.pyx:10455:161: 'c_POSITION_TYPE' is not a type identifier
Traceback (most recent call last):
  File "C:\Users\xxx\xxx\Documents\code\xxx\ZED\zed-python-api-master\src\setup.py", line 194, in <module>
    extList = cythonize(extension, compiler_directives=cython_directives)
  File "C:\Users\joini\AppData\Local\Programs\Python\Python310\lib\site-packages\Cython\Build\Dependencies.py", line 1134, in cythonize
    cythonize_one(*args)
  File "C:\Users\joini\AppData\Local\Programs\Python\Python310\lib\site-packages\Cython\Build\Dependencies.py", line 1301, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: pyzed/sl.pyx

```

---

<div class="post-metadata">

**Author:** ![alassagne](https://community.stereolabs.com/user_avatar/community.stereolabs.com/alassagne/32/2428_2.png) [@alassagne](https://community.stereolabs.com/u/alassagne)\
**Post date:** [September 29, 2023, 2:30pm UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/16 "2023-09-29T14:30:07Z")

</div>

I guess the sources are not up to date. `get_python_api` is the safest way to go, anyway.

To debug your python module, can we go through a few steps ?

- Make sure you use the latest SDK (4.0.7)
- Run ZED Depth Viewer. If it works, the SDK works.
- in the same conda env, or outside, run `python get_python_zpi.py` and then `python <your python script`. Don’t leave the terminal, just to be sure.

---

<div class="post-metadata">

**Author:** ![a11s](https://community.stereolabs.com/letter_avatar_proxy/v4/letter/a/df705f/32.png) [@a11s](https://community.stereolabs.com/u/a11s)\
**Post date:** [September 29, 2023, 2:49pm UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/17 "2023-09-29T14:49:32Z")

</div>

OK, looks like some progress has been made 🙂

It attempts to run Ultralytics now, which is good news, but now throws a new error:

```auto
ImportError: cannot import name '_c_internal_utils' from partially initialized module 'matplotlib' (most likely due to a circular import) (C:\Users\joini\anaconda3\envs\zedyolo\lib\site-packages\matplotlib\ __init__.py)

```

---

<div class="post-metadata">

**Author:** ![alassagne](https://community.stereolabs.com/user_avatar/community.stereolabs.com/alassagne/32/2428_2.png) [@alassagne](https://community.stereolabs.com/u/alassagne)\
**Post date:** [September 29, 2023, 2:50pm UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/18 "2023-09-29T14:50:44Z")

</div>

It’s probably because you have the wrong version of something - can be matplotlib, yolo, etc.

---

<div class="post-metadata">

**Author:** ![a11s](https://community.stereolabs.com/letter_avatar_proxy/v4/letter/a/df705f/32.png) [@a11s](https://community.stereolabs.com/u/a11s)\
**Post date:** [September 29, 2023, 3:14pm UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/19 "2023-09-29T15:14:38Z")

</div>

For some reason there was a series of modules (matplotlib, opencv, pillow) that needed to be uninstalled and re-installed.

Things are looking good thanks! When I run:

```auto
python detector.py --weights yolov8m.pt

```

The following happens in the terminal:

```auto
Intializing Network...
Initializing Camera...
Downloading https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8m.pt to 'yolov8m.pt'...
Initialized Camera

```

and the ZED object detection window opens. However, the window is a black screen and eventually becomes unresponsive.

---

<div class="post-metadata">

**Author:** ![a11s](https://community.stereolabs.com/letter_avatar_proxy/v4/letter/a/df705f/32.png) [@a11s](https://community.stereolabs.com/u/a11s)\
**Post date:** [September 30, 2023, 7:06pm UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/20 "2023-09-30T19:06:39Z")

</div>

Hi, since this has become a new issue, I have set this topic as solved and started a [new thread for the issue with the yolo detector](https://community.stereolabs.com/t/zed-object-detection-yolov8-custom-detector-not-responding/3609).

If you can provide any insight into that, I’d appreciate the help, as since it is not throwing any errors, I have nothing to go on.

Thanks again for your help with the python wrapper etc. 🙂

---

<div class="post-metadata">

**Author:** ![alassagne](https://community.stereolabs.com/user_avatar/community.stereolabs.com/alassagne/32/2428_2.png) [@alassagne](https://community.stereolabs.com/u/alassagne)\
**Post date:** [October 2, 2023, 7:46am UTC](https://community.stereolabs.com/t/zed-with-pytorch-mask-r-cnn/3581/21 "2023-10-02T07:46:06Z")

</div>

Seen that. We’ll continue from there.
