How to evaluate ZED's object detection model using annotated images

[Pettedig]

Hi!

I’m trying to evaluate the performance of ZED’s built-in object detection model (object_medium.3.2.model) on a dataset of pre-annotated images. My goal is to get metrics such as.

• mAP (mean Average Precision)
• Confusion Matrix
• Recall & Precision

As far as I understand, the ZED model only works on live image streams or .svo-files. However, for a proper evaluation, I need to run the model on static images that have corresponding ground truth annotations, which isn’t feasible using the live stream from the camera.

My questions:

1. Is there any supported way to evaluate ZED’s object detection model on a dataset of images with existing annotations?
2. Can I somehow simulate this using a .svo-file created from a video of my dataset?
3. Is it possible to extract per-frame detection results and match them to my ground truth for proper comparison?

I’ve looked through the SDK and the forum, but couldn’t find a clear answer for this use case.

Thanks in advance for any help or suggestions!

[Myzhar]

Hi @Pettedig
Welcome to the Stereolabs community.

@Pettedig

Is there any supported way to evaluate ZED’s object detection model on a dataset of images with existing annotations?

This is not possible because the ZED SDK does not accept input from third-party sources.

@Pettedig

Can I somehow simulate this using a .svo-file created from a video of my dataset?

No, this is not possible.

@Pettedig

Is it possible to extract per-frame detection results and match them to my ground truth for proper comparison?

This is not possible for the same reason.