Extract point cloud data of particular detection and removing background point cloud

varunk12 · April 12, 2023, 8:30pm

I am working on a project with a ZED2 camera. I need help with using mediapipe library hand detection module with ZED2 to extract all points that I get using depth estimation just for the hands and visualize it. can you tell me how can I extract the hand point cloud remove all background point clouds and merge mediapipe hand detection with the ZED2 camera?

alassagne · April 13, 2023, 11:38am

Hello,

I can’t write this code for you. You should check out our Body tracking tutorial : zed-sdk/body tracking/body tracking at master · stereolabs/zed-sdk · GitHub
Using the mask attribute of the BodyData object will give you the pixels involved in a body tracking detection.
Make sure you use BODY_FORMAT_70 to get the detailled hands.

varunk12 · April 13, 2023, 11:41am

Thank You for the help. can you also help me with my another question :
How to extract depth values for particular [x ,y] coordinate depth_value = depth_map_np[y][x]. please look at the post.

alassagne · April 13, 2023, 11:43am

The numpy array contains these values. Check this tutorial zed-sdk/image_capture.py at master · stereolabs/zed-sdk · GitHub which does what you need.

varunk12 · April 13, 2023, 11:45am

i am getting this output:
Depth at (1609,804): [ 3.0693591e-01 1.2300891e-01 5.0059068e-01 -2.3553920e+38] meters
need to understand the values and need a z-axis.

SL-PY · April 14, 2023, 11:59am

how do you get thoses values ?
To get the depth values you shoudl retrieve the MEASURE::DEPTH which is a one channel matrix containing Z values.
It looks like you display Point Cloud values from this sample:

github.com

stereolabs/zed-sdk/blob/master/tutorials/tutorial 3 - depth sensing/python/depth_sensing.py#L70


      
          zed.retrieve_image(image, sl.VIEW.LEFT)
          # Retrieve depth map. Depth is aligned on the left image
          zed.retrieve_measure(depth, sl.MEASURE.DEPTH)
          # Retrieve colored point cloud. Point cloud is aligned on the left image.
          zed.retrieve_measure(point_cloud, sl.MEASURE.XYZRGBA)
          
          
# Get and print distance value in mm at the center of the image
          # We measure the distance camera - object using Euclidean distance
          x = round(image.get_width() / 2)
          y = round(image.get_height() / 2)
          err, point_cloud_value = point_cloud.get_value(x, y)
          
          
distance = math.sqrt(point_cloud_value[0] * point_cloud_value[0] +
                               point_cloud_value[1] * point_cloud_value[1] +
                               point_cloud_value[2] * point_cloud_value[2])
          
          
point_cloud_np = point_cloud.get_data()
          point_cloud_np.dot(tr_np)
          
          
if not np.isnan(distance) and not np.isinf(distance):
              print("Distance to Camera at ({}, {}) (image center): {:1.3} m".format(x, y, distance), end="\r")

In this case the Depth (Z) is the 3rd value. Its unit depends the InitParameters you set.