Concurrent custom Object Detection and Body Tracking

Hello,
using some of your example programs, I’m trying to do custom object detection and body tracking.
This is how I set up both in python.

   positional_tracking_parameters = sl.PositionalTrackingParameters()
    # If the camera is static, uncomment the following line to have better performances and boxes sticked to the ground.
    positional_tracking_parameters.set_as_static = True
    zed.enable_positional_tracking(positional_tracking_parameters)

    body_param = sl.BodyTrackingParameters()
    body_param.enable_tracking = True                # Track people across images flow
    body_param.enable_body_fitting = False            # Smooth skeleton move
    body_param.detection_model = sl.BODY_TRACKING_MODEL.HUMAN_BODY_FAST 
    body_param.body_format = sl.BODY_FORMAT.BODY_18  # Choose the BODY_FORMAT you wish to use

    # Enable Object Detection module
    zed.enable_body_tracking(body_param)

    body_runtime_param = sl.BodyTrackingRuntimeParameters()
    body_runtime_param.detection_confidence_threshold = 40

    obj_param = sl.ObjectDetectionParameters()
    obj_param.detection_model = sl.OBJECT_DETECTION_MODEL.CUSTOM_BOX_OBJECTS
    obj_param.enable_tracking = True
    zed.enable_object_detection(obj_param)

    objects = sl.Objects()
    obj_runtime_param = sl.ObjectDetectionRuntimeParameters()
    bodies = sl.Bodies()

The body tracking part is working, but custom object detection model is not.
Please note that when I comment the body tracking detection, custom object detection is working.

Is custom object detection supposed to work concurrently with body tracking?
Is it working in C++ only?

The only example I found is C++ based and it’s not using custom object detection: zed-sdk/object detection/concurrent detections/cpp at master · stereolabs/zed-sdk · GitHub

Thank you,
Giorgio

1 Like

There seems to be a bug. You should be able to run body tracking and custom object detection concurrently. We’ll make a fix.

Antoine

Hi again,

To use concurrent object detection you need to set different instance_id, like in our sample zed-sdk/object detection/concurrent detections at master · stereolabs/zed-sdk · GitHub

Otherwise it returns an invalid error code. Can you check the error codes ?

Hello,
I set up the instance_module_id for both detector, in this way:

    body_param = sl.BodyTrackingParameters()
    body_param.enable_tracking = True 
    body_param.instance_module_id = 0
    obj_param = sl.ObjectDetectionParameters()
    obj_param.detection_model = sl.OBJECT_DETECTION_MODEL.CUSTOM_BOX_OBJECTS
    obj_param.instance_module_id = 1

I’ve got success in enabling both detectors:

    # Enable Object Detection module
    status = zed.enable_body_tracking(body_param)
    print(f'zed.enable_body_tracking status: {status}')
    if status != sl.ERROR_CODE.SUCCESS:
        print(repr(status))
        exit()
    status = zed.enable_object_detection(obj_param)
    print(f'zed.enable_object_detection status: {status}')
    if status != sl.ERROR_CODE.SUCCESS:
        print(repr(status))
        exit()

this is what I see printed:

zed.enable_body_tracking status: SUCCESS
zed.enable_object_detection status: SUCCESS

I still see the same behaviour:
when both detectors are enabled, just bodies are detected.
If I comment status = zed.enable_body_tracking(body_param) and obj_param.instance_module_id = 1, objects only (because bodies are disabled) get detected.

Giorgio

Hey @alassagne ,
can you confirm if the issue is caused by a bug in zed sdk, or is it something wrong in the way i set up the concurrent detection?

Thanks,
Giorgio

Hi, sorry for the delay. It seems so, but I did not catch it yet. Investigating.

Hi @giorgio
Could you make sure that during the retrieveObjects / retrieveBodies call you’re specifying the correct instance id, similarly to the C++ sample ? This step can be easily missed and default to instance_id 0 which may be the cause of this issue

Hi @adujardin ,
we are calling retrieve_objects and retrieve_bodies in this way:

zed.retrieve_objects(objects, obj_runtime_param)
zed.retrieve_bodies(bodies, body_runtime_param)

As mentioned in this message, we fill obj_runtime_param and body_runtime_param are set up this way:

    body_param = sl.BodyTrackingParameters()
    body_param.enable_tracking = True 
    body_param.instance_module_id = 0
    obj_param = sl.ObjectDetectionParameters()
    obj_param.detection_model = sl.OBJECT_DETECTION_MODEL.CUSTOM_BOX_OBJECTS
    obj_param.instance_module_id = 1

so I believe we are passing the right information to retrieve_objects and retrieve_bodies.

Is this the right way to use ObjectDetectionParameters, retrieve_objects and retrieve_bodies in python?
I had the understanding from previous messages that @alassagne could reproduce the problem but he didn’t manage to fix the bug. Is my understanding correct?

Thanks,
Giorgio

Hi
There are ObjectDetectionParameters and ObjectDetectionRuntimeParameters. ObjectDetectionParameters contains the instance_module_id while ObjectDetectionRuntimeParameters doesn’t, therefore you have to explicitly provide the instance_module_id to the retrieve_objects function (and similarly for Body Tracking).

This gives something like this:

    body_param = sl.BodyTrackingParameters()
    body_param.enable_tracking = True 
    body_param.instance_module_id = 0
    obj_param = sl.ObjectDetectionParameters()
    obj_param.detection_model = sl.OBJECT_DETECTION_MODEL.CUSTOM_BOX_OBJECTS
    obj_param.instance_module_id = 1

    # Enable Object Detection module
    status = zed.enable_body_tracking(body_param)
    print(f'zed.enable_body_tracking status: {status}')
    if status != sl.ERROR_CODE.SUCCESS:
        print(repr(status))
        exit()
    status = zed.enable_object_detection(obj_param)
    print(f'zed.enable_object_detection status: {status}')
    if status != sl.ERROR_CODE.SUCCESS:
        print(repr(status))
        exit()

    obj_runtime_param = sl.ObjectDetectionRuntimeParameters()
    body_runtime_param = sl.BodyTrackingRuntimeParameters()

    [...]

    zed.retrieve_objects(objects, obj_runtime_param, obj_param.instance_module_id)
    zed.retrieve_bodies(bodies, body_runtime_param, body_param.instance_module_id)

I think this was the source of the issue. From my testing, there were no issues with this parameter but it can be easily missed with its default value, including when @alassagne tried to reproduce the behavior.

We’ll work on adding more samples, especially in Python to improve the clarity.

Let us know if the snippet above doesn’t provide the expected behavior

1 Like

Hi @adujardin ,
I tried the snippet you sent, I’m reposting it here to make sure everything is correct:

positional_tracking_parameters = sl.PositionalTrackingParameters()
# If the camera is static, uncomment the following line to have better performances and boxes sticked to the ground.
positional_tracking_parameters.set_as_static = True
zed.enable_positional_tracking(positional_tracking_parameters)

body_param = sl.BodyTrackingParameters()
body_param.enable_tracking = True                # Track people across images flow
body_param.enable_body_fitting = False            # Smooth skeleton move
body_param.detection_model = sl.BODY_TRACKING_MODEL.HUMAN_BODY_FAST 
body_param.body_format = sl.BODY_FORMAT.BODY_18  # Choose the BODY_FORMAT you wish to use
body_param.instance_module_id = 0


obj_param = sl.ObjectDetectionParameters()
obj_param.detection_model = sl.OBJECT_DETECTION_MODEL.CUSTOM_BOX_OBJECTS
obj_param.instance_module_id = 1

status = zed.enable_object_detection(obj_param)
print(f'zed.enable_object_detection status: {status}')
if status != sl.ERROR_CODE.SUCCESS:
    print(repr(status))
    exit()

status = zed.enable_body_tracking(body_param)
print(f'zed.enable_body_tracking status: {status}')
if status != sl.ERROR_CODE.SUCCESS:
    print(repr(status))
    exit()

obj_runtime_param = sl.ObjectDetectionRuntimeParameters()
body_runtime_param = sl.BodyTrackingRuntimeParameters()
body_runtime_param.detection_confidence_threshold = 40

objects = sl.Objects()
bodies = sl.Bodies()

[...]

zed.retrieve_objects(objects, obj_runtime_param, obj_param.instance_module_id)
zed.retrieve_bodies(bodies, body_runtime_param, body_param.instance_module_id)
print(f'objects: {objects.object_list}')
print(f'bodies: {bodies.body_list}')

In this case I can only retrieve bodies.body_list, while objects.object_list is empty.
If I keep the exact same code and I swap the instance_module_ids, such that I have:

body_param.instance_module_id = 1
obj_param.instance_module_id = 0

I have the opposite behaviour, so I can only retrieve objects.object_list while bodies.body_list is empty.

So basically, I can only retrieve objects or bodies that have instance_module_id == 0.

Please, let me know if I’am still missing anything or if there is actually a bug in the python wrapper.

Thanks,
Giorgio

Ok, thanks for the detailed report, it looks correct.

Just in case here’s a full sample with Body tracking and Object detection in Python that is working correctly as is GitHub - adujardin/zed-concurrent-detections-python-proto: Prototype of ZED example with Concurrent detection Body Tracking / Object Detection, I tested it with the latest release (4.0.4). Could you test it ? It may be specific to the Custom box input if the sample is working