How to create top down (birds eye) image from point cloud?

My goal is to use my Zed 2i camera that is mounted at an angle to produce an orthographic top down view of the point cloud. Ultimately I want this view to provide a 2D opencv image that can be used for basic navigation.

Currently I am accomplishing this by creating a secondary 2D point cloud that is comprised only the x, y, and color values. Due to iterating through every single pixel this takes a long time to render.

I have attempted to rotate the point cloud using the Euler-Rodrigues formula and then use the provided 3D points to 2D pixels formula but had similar results regarding computing time.

I’m curious if anyone on here can think of a more efficient approach or if there is a built in functionaility within the SDK that I am missing.

Thank you.

Hi and welcome to our forum !

Running algorithms on big matrix can always be compute-expansive if not done with care. I’d suggest :

  • Try do do as few copies as possible. Always use reference and pointers when you can
  • Try to make you program multithreaded. For example, you can easily achieve that in C++ with openmp
  • Make non-blocking functions that run in the background, with threads.

Antoine

Hey Antoine and thank you for the response!

I’ll give that a shot and report back in time. Another option that I found recently is using the top down orthographic view from rviz. Certainly far from optimal but I’m hoping I can use that view to scratch together a prototype environment viewer.

FWIW I was looking into that as well.

One hacky workaround I came up with is using Open3D’s project_to_depth_image.

Notice it’s open3d.t.geometry.PointCloud (the tensor/GPU accelerated structure), not open3d.geometry.PointCoud (CPU implementation).

I was trying that in Python and got decent frame rates. It wasn’t a 100% super accurate orthorgraphic view, but close enough (tweaking the camera matrix).

I had some headaches compiling the Open3D Python bindings from source with CUDA support to get the speedup (the CPU version is super slow). You can find my notes on stackoverflow.

It’s worth remembeing that (AFAIK) the ZED Python API only has CPU support and when visualising OpenCV (default pip install) that’s also CPU which is a far from ideal CPU <> GPU conversion (CPU point cloud to GPU point cloud to depth mat and back to CPU)

It haven’t tried this workaround with the C++ API, but in theory, you could get the ZED SDK point cloud on the GPU directly, move/wrap the data to an Open3D GPU PointCloud, project to a depth image using the camera matrix to get the top view, then process as needed.

Hope this helps and I’m interested in seeing other ideas.

1 Like

I’m struggling with the concept of creating a bird’s eye view perspective. I’ve created a simple 2D BEV that takes into consideration the camera distance to the bounding box of the user, but I would like to understand a bit more about how it would work with a point cloud.

Is it possible to provide more hints on how to do it or share a sample? Thanks in advance.