Within Unity, I managed to get the 2D & 3D Object Detection module to work but I’d want to replace the object detection module with a custom one (either just replacing the weights of the networks or change the network itself with different classes, different architecture, …).
There is the option within Unity to take a CustomObjDetection model, but it does not output/visualize any boxes. I assume the corresponding model has to be trained/built first. Is this correct?
I found
Do I have to follow that?
Ideally, I want to program in Python, but probably I will have to switch to C#, correct?
It would be nice if you could guide me to where I would have to start working on?
Hi,
There is a example scene called “CustomObjectDetection” next to the 3D and 2D ones. There is some information in the Readme available next to the scene as well.
It uses yolo from the Opencv integration as a example. You can start from this sample and see how it goes.
Do not hesitate to ask for help if needed.
Best,
Benjamin Vallon
Thanks for your reply!
Do you mean the Radme.md in Assets/ZED/Examples/Object Detection?
Hi,
Yes. There is not much information but it should be enough to at least run the sample and understand how it works. You can also directly look the OpenCV example.
Stereolabs Support
Hi Benjamin,
I downloaded OpenCV for Unity and downloaded all the models according to the setup_dnn.pdf.
I changed the Object Detection Model to “CUSTOM_BOX_OBJECTS” expecting that it automatically somehow calls the ZEDCustomObjDetection.cs file.
However, I obtain compile errors, namely “error CS0117: ‘Aruco’ does not contain a definition for ‘DICT_APRIL TAG_36h11’”.
This is inside ZED/Examples/OpenCV ArUco Detection/Scripts/Core/ZEDArUcpDetectionManager.cs(252,37).
(in fact all the dicts throw compile errors)
Do I need to compile the example from zed-examples/object detection/custom detector/cpp/opencv_dnn_yolov4 at master · stereolabs/zed-examples · GitHub first or is that unrelated?
I saw that you refer to Aruco markers. I probably have to download these markers to somewhere, right?
The link in the Github https://raw.githubusercontent.com/opencv/opencv_contrib/master/modules/aruco/src/predefined_dictionaries.hpp yields error 404 though. Any other way to go about that?
This issue has been solved with the last commit of the ZED-Unity plugin (this one : https://github.com/stereolabs/zed-unity/commit/307ae260b6c7d8d1c258dec4623832e5c4121af8).
Can you try to use the latest version of the plugin ?
Best,
Benjamin Vallon
Stereolabs Support
Thanks, these lines fixed the missing-definition-error.
Now, when running the scene with the CUSTOM_BOX_OBJECTS on (Examples/Object Detection/Scenes/Custom Object Detection.unity), it says NullReferenceException inside ZEDCustomObjDetection.
Do I need to set any paths?
I’m not sure what I need to adapt
The first errors are from you, saying that I should download everything as indicated in setup_dnn_module.pdf. I downloaded all models though, and inside OpenCVForUnity/dnn, I see the same files as in the screenshot.
Ok, when adjusting the paths to absolute paths inside Unity for Model, Config and Classes, it runs without errors, but it is a little bit laggy and does not detect anything (at least nothing is visualized while me standing&sitting infront of the camera). I assume the weights are already pretrained, right?
Furthermore, this way of detecting stuff is independent of the tutorial here, right?:
Just tested it on Windows, the custom object detection model is running. But it only works for the 3D Object Visualization. For 2D is outputs random boxes.
THe OpenCV examples work though. How would I have to adapt the custom obj det scene to work for 2D objs?
Finally, got it to work by just copying the 3D scene and changing the 3D Object Visualizer to 2D Object Visualizer. But the lagginess remains. The OpenCV example is much smoother despite running the same model. Don’t know why…
