the pointcloud is correct, but the masked image is wrong.
I suspect that the plane behind the cone is being considered and thus the masked image contains the cone
Hi @kuku929,
The plane detection sample does not use depth information to display the overlay under or behind objects.
As this is a simple example of the use of the plane detection algorithm, the plane is displayed in the 2D image.
I see. Is there a way I can get a mask which “ignores” points not visible in the image?
can I get a map from the 2D (u,v) image coordinates to the 3D coordinates and just use those points for the plane detection.
Hi @kuku929,
I believe what you are trying to do is change the OpenGL rendering depending on the depth map computed by the ZED SDK.
To perform the effect you are looking for, you can update the OpenGL viewer from the sample: zed-sdk/plane detection/plane detection/cpp/src/GLViewer.cpp at master · stereolabs/zed-sdk · GitHub
to take into account the distance to draw or not the pixel if there is an object closer to the camera than the plane.
The distance is already included in the vertex and fragment shaders (zed-sdk/plane detection/plane detection/cpp/src/GLViewer.cpp at master · stereolabs/zed-sdk · GitHub), so you can add a rule stating “if the plane vertex is further away than distance, do not display the point”
makes sense. I am new to computer graphics so am a bit confused about the implementation.
- Are you implying something like depth testing?
- The cone or any object for that matter is not part of the mesh that is being rendered by the MESH_FRAGMENT_SHADER, so how can I get their depth values?
from what I understand, the problem is in the plane detection algorithm, which returns points behind objects as part of the mesh.
Hi @kuku929,
The Plane Detection algorithm matches the points that belong to the same plane in 3D, based on the 3D point cloud and a pixel in the image.
The resulting plane is a surface in 3D, and the mesh from the extractMesh method returns a mesh representation of this plane that can be used for different applications. For this reason, we do not filter out objects “in front of” the mesh, as this is dependent on the visualization you wish to have for your application.
What I would recommend is to retrieve the depth image from the ZED SDK, load it in OpenGL as a 2D texture, and filter out all points of the mesh that are “behind” the distance value given by the depth texture.