I’ve been looking for a way to be able to insert a depth capture of the ZED2 into the scene in Unity, similar to the green screen example in the docs, but only on an area that a person is detected (due to lack of a proper green screen).
First, for this type of application, I recommend using the NEURAL depth mode. This mode gives the most accurate and smooth depth map which is very important for this type of application.
There are multiples ways of doing it :
The most “basic” one is to simply “clamp” the depth map to a certain threshold. All the parts of the image that are closer than this threshold are visible, the rest are hidden. For that, you can use the max depth range parameter available in Unity, on the ZED Manager script, in the Advanced settings section. With this, the depth range is set once and is not changing over time.
Then, you can improve this first method by automatically modifying this depth value to follow a person in the scene. To do that, you can use the Object detection module available in the ZED SDK to track the 3D position of a person in the scene and adapt this depth range over time.
I think you can start by testing those solutions that are relatively basic and will not probably work well in some cases but might be enough for you at first. It can also give you ideas on how to improve it for your specific use case in the future.