Unity custom Object Detection with *.pt file

Software
1

Hi everybody,

I am trying to figure out how to use Unity for running a custom object detection model. I have trained a custom Yolov8 model , resulting in a *.pt file. How can I use the Unity SDK to use this file?

Thank you so much in advance,

2

Hello @wimvanhenden-tool, welcome to the forums!

We have a sample using OpenCV inside the project, with a readme next to it explaining how to get it running.

You can find some more info from this post:

Feel free to ask any question you have along the way :slight_smile:

3

Hi JPlou,

Thank you so much for your answer. So when using Unity we need to use OpenCV as a backbone, understood.

  1. Does that mean we need to run Yolov8 models (*.pt files) through OpenCV for using them in Unity?
  2. I sthere any documentation on how to run custom models through OpenCV in Unity to then connect it to the Zed Unity SDK?
  3. Does this account for extra delay, possible performance issues?

Thank you so much,

Thank you so much.

4

Hello @wimvanhenden-tool,

Using OpenCV as a backbone is how we did our sample in Unity. The dedicated script is located here: ZEDCustomObjDetection.cs
That being said, you can implement the Custom Object Detection the way you want, in the end you have to make the SDK ingest bounding boxes.

  1. Does that mean we need to run Yolov8 models (*.pt files) through OpenCV for using them in Unity?

To use our sample, yes.

  1. I sthere any documentation on how to run custom models through OpenCV in Unity to then connect it to the Zed Unity SDK?

Not more than the Readme and the comments in the script linked above, sorry.

  1. Does this account for extra delay, possible performance issues?

We haven’t measured that on our side, but it’s possibly less optimized than running a C++ or Python custom detector sample.

I hope this helps :slight_smile: