Object Misclassification (With Custom Detector)

Software Python
,
1

Hi there,

I am running my cameras with object detection using CUSTOM_YOLOLIKE_BOX_OBJECTS, NMS3D_PER_CLASS, and a custom onnx file. I retrieve_objects() into an sl.Objects() variable, and most of the time everything works just fine. Sometimes, however, I get a “misclassification".

For example, my custom detector detects people and pylons. When a misclassification occurs, it labels a person as a pylon instead of a person. And not just for a frame or two. That misclassification lasts until the person exits the frame.

Now don’t get me wrong, I am well-aware that this could be a model error. But I’ve seen it with multiple models and believe that the ZED SDK (v5.0) output could potentially be outputting the incorrect raw_label. Perhaps its because of the way I configured it or am retrieving the objects which is causing some undefined behavior.

Here is a more in-depth look at my configuration:

detection_parameters = sl.ObjectDetectionParameters()
detection_parameters.detection_model = sl.OBJECT_DETECTION_MODEL.CUSTOM_YOLOLIKE_BOX_OBJECTS
detection_parameters.custom_onnx_file = "my_model.onnx"
detection_parameters.custom_onnx_dynamic_input_shape = sl.Resolution(608,608)
detection_parameters.filtering_mode = sl.OBJECT_FILTERING_MODE.NMS3D_PER_CLASS
detection_parameters.allow_reduced_precision_inference = True
detection_parameters.enable_tracking = True
detection_parameters.enable_segmentation = False

And then I retrieve objects like this:

_objects = sl.Objects()
_rt_params = sl.ObjectDetectionRuntimeParameters()
_rt_params.detection_confidence_threshold = 30

zed.retrieve_objects(_objects, _rt_params)
object_list = _objects.object_list

And then I get the classes like this:

[det.raw_label for det in object_list] # Gives the class ID

Then I can get my class name using my own Python dictionary that maps ID → label.

Has anyone else had this error, or does anyone see anything wrong with my configuration?

Please let me know if you’d like some more info.

Thanks!

2

Hi @stripedapus
Welcome to the Stereolabs community.

To debug the problem that you described, we need:

  • SVO files recorded in the same conditions (you can use ZED Explorer) that cause the same problem when used as input
  • The custom model that you are using
  • Your source code

You can send them to support@stereolabs.com if you do prefer not to share them publicly.

Can you also verify if using “external inference” (C++ and Python examples), the problem persists?

3

Hi @Myzhar ,

Thanks for your response.

I’ve just discovered more information pertaining to the issue.

I changed to simply using yolo12n.onnx (built straight from the Ultralytics yolo12n.pt trained on the COCO dataset) so that I could confirm the issue was not my model and so that I could have many classes and sure enough - the issue was still there. Usually after some time running (~5min) when I would walk into frame I would be misclassified (ex. cell phone or dining table). When I exited and entered the frame, I would be misclassified again as something else (ex. chair or tv).

The fact that I was consistently being misclassified led me to believe it may be a tracking issue, and sure enough, when I changed my sl.ObjectDetectionParameters object to have .enable_tracking = False, I no longer saw the issue.

Of course, this comes with drawbacks. Without object tracking, if an object is not detected for a frame or briefly occluded the detection disappears (with object tracking enabled it seems to remain). This could be positive or negative depending on your use case, for me it’s a negative.

My best guess would be that the object tracking may incorrectly identify an object in the first frame (perhaps the classes had close confidences but the incorrect one was slightly higher) so in that frame the incorrect class is chosen. But then with object tracking enabled the class can’t be changed so it remains incorrect. That’s just a hypothesis, though.

Nevertheless, turning off tracking seemed to resolve the misclassification issue. Perhaps some future work can be done to prevent this in the future.

TL;DR:
Solved misclassification issue by doing:

detection_parameters = sl.ObjectDetectionParameters()
detection_parameters.enable_tracking = False

Thanks!

4

Thank you for the useful details.
The problem has been reported to the SDK Team and logged.
It will be analyzed and fixed with one of the next versions of the ZED SDK.

If you can record SVO files that allow to replicate the issue, please share them with us adding source code that you use and the ONNX model.
You can send them to support@stereolabs.com.

The more information you can provide, the more it will be easy to replicate and fix the issue.