I’ve spent some time to try to run and optimize custom object detectors inside Unity using either ZED2’s Unity Plugin or just OpenCVforUnity.
Took me a bit of time to notice, but I think the problem is:
Neither approach is utilizing the GPU really for inferencing the networks. I’m not too sure about the ZED examples, but the OpenCVforUnity ones definitely don’t.
That’s a huge downside and renders most object detectors useless. Am I missing something or is that assessment correct?
If so, I’m considering different options to utilize the GPU:
-Barracuda (Introduction to Barracuda | Barracuda | 1.0.4)
-Unity-TensorRT (GitHub - aman-tiwari/Unity-TensorRT: Unity + TensorRT integration)
-NatML (NatML for Unity - Unity)
I don’t have any experience with these frameworks but I fear that it will be quite unstable or hard to adapt to custom model to use these. Do you have any experience and/or tipps regarding them?
At this point, I’m a bit disappointed in Unity generally when it comes to neural network inference. I’m considering streaming the video using the ZED SDK. However, I’m a bit afraid of latency and synchronization issues.
Therefore, I wondered whether you have any tipps wrt sending the output of the neural network. Can I package it somehow in the output stream? If so how?
Thanks in advance!