I’m trying to extract point cloud base on 2D mask, but cannot get them to align nicely. Here are some screenshots:
The bottom half is used for masking, where the green border is the mask outline.
The top half is my point cloud - green points are outside the mask; white points are inside the mask; red points are inside the mask but they shouldn’t. When you look closely you can see a fine green edge on the left side of the silhouette, that edge should in fact be white.
I’m using the formula on this page to map my 2D mask to the point cloud.
The visualization is done on opengl, and here’s the vertex shader:
#version 330 core
layout(location = 0) in vec4 in_VertexRGBA;
uniform mat4 u_mvpMatrix;
uniform sampler2D mask;
uniform vec2 cam_res;
uniform float f_x;
uniform float f_y;
uniform float c_x;
uniform float c_y;
out vec4 b_color;
void main() {
vec3 position = in_VertexRGBA.xyz;
float u = (( position.x / position.z ) * f_x + c_x);
float v = (( position.y / position.z ) * f_y + c_y);
vec2 uv_mirror = vec2(u, v) / cam_res;
vec2 uv = vec2(1. - uv_mirror.x, uv_mirror.y);
vec4 maskPx = texture(mask, uv);
b_color = vec4(vec3(255.0f), 1.);
gl_Position = u_mvpMatrix * vec4(position, 1);
gl_ClipDistance[0] = float(maskPx.r < 0.5);
gl_ClipDistance[1] = float(position.z < 2);
}
Then in my fragment shader, I set the color base on gl_ClipDistance[0] and gl_ClipDistance[1]
My uniform values come from:
- cam_res: (2208, 1242) for HD2K
- fx, fy, cx, cy:
camera.get_camera_information().camera_configuration.calibration_parameters.left_cam - mask: my bitwise 2D mask
Meanwhile I retrieve the image, depth map, and point cloud using the following calls
camera.retrieve_image(img_mat, sl.VIEW.LEFT, sl.MEM.CPU, res)
camera.retrieve_measure(dep_mat, sl.MEASURE.DEPTH, sl.MEM.CPU, res)
camera.retrieve_measure(pc_mat, sl.MEASURE.XYZRGBA, sl.MEM.CPU, res)
Here are a few more samples for reference:
Looking forward to any help!