Using SDK person detections and custom detector at the same time

Software Python
1

Hi,

I would like to use the Zed camera to detect different objects in the images at the same time (e.g. persons and my custom objects). in the Zed SDK it is possible by default to detect e.g. persons. I already use Detectron2 to detect my custom objects and now I’ve seen here that I can feed my results from Detectron2 into the Zed SDK in order to then calculate and visualize the relevant data (depth, etc.) in the SDK.

I saw that the detection model in the SDK for the custom objects has to be set like this:

detection_parameters.detection_model = DETECTION_MODEL::CUSTOM_BOX_OBJECTS;

My question is:

Is it possible to detect/show my custom objects and persons from the standard SDK in the images at the same time?

2

Hi @JoeHhuio98
Welcome to the Stereolabs community.

The new ZED SDK v4 allows to instantiate different AI models simultaneously by simply creating different Object Detection objects and assigning a different instance_module_id to each of them. The same instance_module_id must be used when calling the retrieveObjects function to retrieve detection data.

3

Ok, glad to hear that these different objects can be detected at the same time :slight_smile: thanks for the info!

Did I understand correctly that the functionality as described by you here remains the same in SDK v4:

Workflow:

After each grab call, the image can be retrieved and sent to your detector, the bounding box detections result can be ingested into the ZED SDK for processing. Then using retrieveObjects the 3D tracked objects can be retrieved.

That means I get the bounding box results (coordinates) back from my custom detector and these are then transferred directly to retrieve_objects (for the correct instance_module_id of course)? Or is there something else important that needs to be taken into account?

Thanks again

4

No, the retrieveObjects does not change, you must only add the correct instance_module_id that you used in initialization

5

ok, but what data does the ZED SDK v4 expect from my custom objects exactly (from detectron2 in this case)? Are the bounding boxes per image enough? I do not quite understand the connection between the output of my custom objects (returned bounding boxes) and retrieve_objects …

6

You can follow these examples:

7

@Myzhar ok thank you very much. This looks promising.

Just two more questions:

1.) I’ve seen here that people had problems in the past using detectron2 and zed becuase both use CUDA at the same time. Is this why there is a new sample in the current SDK called “concurrent detections” beside “birds eye viewer”, “image viewer” and “custom detector”?

2.) Is it also possible to use a custom detector via ROS and zed?

8

No, concurrent detection is a new feature introduced in ZED SDK v4 to be able to run different ZED detectors simultaneously. The GitHub thread that you linked contains the correct solution for your issue.

This is not currently available. We will add this feature to the ZED ROS 2 Wrapper in the next few weeks.

9

Hi @Myzhar
I guess it’s best to have the sdk’s object detector and the custom detector each in their own thread, where in the subroutine for each thread retrieve_objects is called with the appropriate instance_module_id, right?