Using 3D Sensor for Behavioral Study in Library – Setup, Accuracy & Tracking Question

Hardware
1

Hello everyone,

I’m currently designing an experiment using the Stereolabs 3D sensor to study behavioral interactions in a library setting, and I would appreciate your input regarding hardware and setup feasibility.

The goal of the study is to detect and record when a person approaches a bookshelf and either takes a book or puts one back. To respect GDPR guidelines, I will not store video footage, but instead rely on skeleton tracking to capture movement data. I plan to define regions of interest (ROIs) corresponding to the bookshelves, and monitor interactions between hand keypoints and these ROIs.

So far, I am using the Robotic Perception Kit, and I’m able to detect interactions based on hand-ROI intersections. However, I’m facing the following challenge:

How can I distinguish between a person taking a book vs. putting one back?

Additionally:

I am considering adding an extra camera to improve accuracy, coverage, and robustness of detection. Would that help, and is it supported within the existing SDK and hardware ecosystem?

Is there any way to identify or tag a specific individual within the environment? For example, if I want to track only my participant during a pilot test, can I use a visual marker such as a specific color jacket, a yellow hat, or a wearable tag/object to allow the system to recognize and differentiate them from other people?

Thanks so much for your support,

Best regards,

Lio Nord

2

Hi Leo,

Thank you for reaching out to us.

Using multiple cameras to cover more area or increase the tracking accuracy is indeed a feature available in the ZED SDK called “fusion”. You can find more info here: https://www.stereolabs.com/docs/fusion/overview

It only requires the cameras to slightly overlap in order to be able to calibrate the system.

How can I distinguish between a person taking a book vs. putting one back?

I think you need to pair the skeleton tracking with another AI model that will detect books. Then, you can verify if there is a book in the hand of the person of interest.
The “book detection” is not supported in the ZED SDK, so you might need to use an external ai model and take advantage of our custom detector feature for this.

Is there any way to identify or tag a specific individual within the environment? For example, if I want to track only my participant during a pilot test, can I use a visual marker such as a specific color jacket, a yellow hat, or a wearable tag/object to allow the system to recognize and differentiate them from other people?

No, this feature is not available in the ZED SDK. You might need to use another tracking system with a Aruco code or something similar.

Stereolabs Support