Timeout detected on camera [FUSION] and empty camera metrics

Software Python
, ,
1

We have a fusion setup involving 3 cameras, running SDK5.0 EA. We managed to calibrate the setup in ZED360 and can run our fusion script. However, even though it detects bodies, they are quite jittery and get reassigned with new IDs sometimes while in the middle of the tracked area. I was wondering if maybe not all cameras participated in the fusion so I tried to set up some metrics.

My code looks like this

metrics = fusion.get_process_metrics()[1].camera_individual_stats
for key in metrics:
print(metrics[key].is_present)
print(metrics[key].synced_latency)
print(metrics[key].received_fps)
print(metrics[key].delta_ts)

is_present is True for all cameras but the other values are all 0.0

When entering the loop, fusion also gives me this error once, supporting my idea of missing cameras in the fusion and I don’t know what to do with that

[ZED][WARNING] Timeout detected on camera [FUSION]. The camera will be removed from the synchronization until new data is available.

What might cause these issues?

2

Are you in python?

Can you try to use the c++ sample instead?

Thanks!

Stereolabs Support

3

Hi Benjamin,

I’m in Python, yes, but also found the embarrassing solution . I called metrics on.y once before entering the loop. Guess it was just too late yesterday…

Now my only issue is the jittery body tracking. I’m running my senders with these parameters on NEURAL_LIGHT

body_param = sl.BodyTrackingParameters()
body_param.enable_tracking = False
body_param.enable_body_fitting = False
body_param.detection_model = sl.BODY_TRACKING_MODEL.HUMAN_BODY_MEDIUM
body_param.body_format = sl.BODY_FORMAT.BODY_34
body_runtime_param.detection_confidence_threshold = 10

Changing to ACCURATE on the body tracking model didn’t do much of a difference besides pulling us down to 15fps on the ZEDbox senders. Our use case is tracking people running, which currently seems not possible with this setup. Do you have any advice how to improve this?

Should it be possible to track running people in general?

4

Can you confirm the calibration has been done correctly?

You can load your calibration file in ZED360. There, you should see the skeleton detected from each camera.
All skeletons should be aligned.

Stereolabs Support

5

When I calibrate everything, the skeletons are all well aligned after we changed camera positions a bit compared to yesterday. In the older setup they were well alined as well but had very strong distortion. The fused body is still jumping and getting new ID assigned in our current setup.

If I load the calibration into ZED360, the cameras are all shown in the same position as if they were not calibrated (we currently back on SDK 4.2.5 now, since we wanted a comparison).

Can you also confirm, if calling the single bodies from individual cameras should provide joint data? I can call the body ID and tracking state, but body.local_position_per_joint gives just an empty array even if the state is OK.

6

https://share.plan-valley.one/share/HfrM1HOKCEaL

This is how calibration and calibrated setup looks in ZED360

7

edit: We went further down back to SDK 4.1.4 where the calibration file had the old format and the whole setup works as expected now. Still wondering what caused the issues.

We just ran the cpp fusion sample to have a look at the avatars and it seems like we have the same issue as when opening the calibration file in ZED360. The fusion behaves as if the three cameras were in one position, seeing three different people instead of one fused. So I assume something with the calibration is off, but there are definetely values in it that look correct.

The file looks different than the sample in the documentation and the format I remember from earlier SDK releases, though. Might this be our underlying issue?

{
    "30761362": {
        "FusionConfiguration": {
            "communication_parameters": {
                "CommunicationParameters": {
                    "communication_type": "LOCAL NETWORK",
                    "ip_add": "10.0.110.47",
                    "ip_port": 30000
                }
            },
            "input_type": {
                "InputType": {
                    "input": "",
                    "input_type_conf": "0",
                    "input_type_input": "LAST"
                }
            },
            "override_gravity": false,
            "pose": "1,000000 0,000000 0,000000 0,000000 0,000000 1,000000 0,000000 0,227987 0,000000 0,000000 1,000000 0,000000 0,000000 0,000000 0,000000 1,000000",
            "serial_number": 30761362
        }
    },
    "32439504": {
        "FusionConfiguration": {
            "communication_parameters": {
                "CommunicationParameters": {
                    "communication_type": "LOCAL NETWORK",
                    "ip_add": "10.0.110.48",
                    "ip_port": 30002
                }
            },
            "input_type": {
                "InputType": {
                    "input": "",
                    "input_type_conf": "0",
                    "input_type_input": "LAST"
                }
            },
            "override_gravity": false,
            "pose": "0,710427 -0,028563 -0,703192 0,957363 -0,033087 0,996716 -0,073914 0,340607 0,702993 0,075777 0,707148 1,946823 0,000000 0,000000 0,000000 1,000000",
            "serial_number": 32439504
        }
    },
    "34917315": {
        "FusionConfiguration": {
            "communication_parameters": {
                "CommunicationParameters": {
                    "communication_type": "LOCAL NETWORK",
                    "ip_add": "10.0.110.50",
                    "ip_port": 30006
                }
            },
            "input_type": {
                "InputType": {
                    "input": "",
                    "input_type_conf": "0",
                    "input_type_input": "LAST"
                }
            },
            "override_gravity": false,
            "pose": "-0,223202 -0,088873 -0,970712 1,767578 -0,055181 0,995390 -0,078444 0,382958 0,973209 0,036056 -0,227077 3,666693 0,000000 0,000000 0,000000 1,000000",
            "serial_number": 34917315
        }
    }
}
8

The format has slightly changed, but as long as the ZED SDK managed to open it, it should be fine (otherwise, it would give you an error).

You need to calibrate your system with the same version you are using to run the body tracking sample, otherwise you might run into format incompatibility issues

Stereolabs Support

9

We had always used the same SDK version on all the involved computers. Calibration had worked, it generated a proper file and when loading it inbZED360 immediately after that, it gave the cameras in zero position again as if nothing ever happened.

As mentioned in my other thread, I’m also having issues with the residual and Jacobian evaluation failure, which happened in several SDK versions and is giving me a hard time at the moment.