Segmentation masks not available using custom Object Detection with YOLOv8

Software Python
1

Hi there,

I’m using:

  • ZED SDK 4.1
  • ZED X camera

and trying to get the mask output from a YOLOv8 customer detector.

I’m using the ZED SDK custom detector example, with object mask computation enabled:

obj_param = sl.ObjectDetectionParameters()
obj_param.detection_model = sl.OBJECT_DETECTION_MODEL.CUSTOM_BOX_OBJECTS
obj_param.enable_tracking = True
obj_param.enable_segmentation = True  # masks enabled
zed.enable_object_detection(obj_param)

and a printout of the mask information:

for obj in objects.object_list:
    if obj.tracking_state == sl.OBJECT_TRACKING_STATE.OK:
    print(obj.mask.is_init())
    print(obj.mask.get_data())

However, the result is:

  1. obj.mask.is_init() always returns False
  2. obj.mask.get_data() always returns []

This post seems to suggest that segmentation masks are available with custom object detectors (e.g. YOLOv8) via the ZED SDK, however this post suggests they aren’t available.

Here’s the complete code:
custom_detector_w_masks.py (7.7 KB)

Is it possible to retrieve instance segmentation masks using the ZED SDK Object Detection model and a custom object detector, e.g. YOLOv8?

Thanks!

2

Hi @andrew.stringfield,

Currently, the ZED SDK only supports ingesting custom bounding boxes from a custom object detector model.

As Object Detection models are increasingly supporting segmentation masks, we are currently working on supporting custom instance segmentation masks in the ZED SDK to add our 3D tracking capabilities. This is planned for future versions of the SDK.

Both posts you’ve linked are correct: the SDK can ingest custom bounding boxes, and if segmentation is enabled, the SDK computes a segmentation mask based on the geometry of the object (not with AI).

With the errors you’ve noticed, we have indeed detected an issue in the return of the masks, and are preparing a fix for a patch of SDK.

3

@mattrouss thanks for the update!

When you say:

the SDK computes a segmentation mask based on the geometry of the object

How is this done? Is the mask calculated as a square that’s the same area as the bounding box, or something else?

Thanks!

4

Hi @andrew.stringfield,

Indeed this is a limitation of the custom object detection segmentation. As we do not have additional information on the objects, it is difficult to refine the mask of the object in a generic way.

For comparison in the ZED SDK Object Detection classes which are known, you can retrieve the segmentation masks, and this is performed using the geometry of the object, a.k.a its point cloud information in 3D.

So in your case, ingesting custom segmentation masks is probably more interesting, this is a feature in our roadmap short-term.

5

Hey @mattrouss, I’m also looking for a way to do this, is there any update your side on when it might make it’s way into the SDK?

6

Hi @andy,

Yes, the method was implemented in ZED SDK 4.2.

You have a C++ sample here that shows a usage with Yolov8 instance segmentation, there is also an equivalent Python sample

7

Brilliant, I didn’t spot that, I can see the model I have working in the python example there, but if I load the same model into the Unreal Plugin I don’t see the same masks generated. Since I don’t see any mention “ingestCustomBoxObjects” or “ingestCustomMaskObjects” in the Unreal Plugin I’m guessing this isn’t implemented there?

8

That is correct it is not implemented, you can create an issue on the repository directly.

9

Even though the fonction has not been added in the Unreal Project, it is available in the C wrapper that is used to interface with the ZED SDK (here : zed-UE5/Plugins/Stereolabs/Source/ThirdParty/sl_zed_c/include/sl/c_api/zed_interface.h at main · stereolabs/zed-UE5 · GitHub), so, in theory, calling this function should work.
You can create a new method in USlCameraProxy that calls this C function :
zed-UE5/Plugins/Stereolabs/Source/Stereolabs/Private/Core/StereolabsCameraProxy.cpp at main · stereolabs/zed-UE5 · GitHub

I should warn you that it has not been tested on our side.

10

That’s useful thanks, I did look through USlCameraProxy to see if this was a simple fix but it looks like you need to run the object detection algorithm externally to the ZED SDK and then feed those objects in, rather than being able to simply enable this using the onnx that is selected in the Unreal Plugin?

11

To use a custom onnx file, set the OBJECT_DETECTION_MODEL to CUSTOM_YOLOLIKE_BOX_OBJECTS(sl Namespace Reference | API Reference | Stereolabs) and provide the path to the onnx file with ObjectDetectionParameters::custom_onnx_file (ObjectDetectionParameters Class Reference | API Reference | Stereolabs)