Changing people's background

Software Unity
1

Hi all,

I am a total newbie with Unity and I need some kind of direction on where to head in my current project.
Goal is to identify people, remove them from the image and place them to a new virtual background which will have layers.

I got some results with Python but I am totally clueless with Unity.

I have been trying to work with getting segmentation to work but I noticed that it is a “safety feature” that people’s arms are not included in the segmentation mask.

In a nutshell I need to put people into a plane/canvas and then I can place them in a 3D environment to where ever I need to in the scene and in the scene there will be other game objects.

Can anyone point to some example where whole people have been masked out and how to put those pixels to a new game object’s texture? Or maybe give some idea how to approach this dilemma?

2

Hi @tuorpete, welcome to the forums!

I suggest following the same method as our example on Unreal: Background Subtraction with UE5 - Stereolabs

It should be possible with Unity too, to basically clamp the depth.
You should find the needed parameters in the ZED Manager component.

3

Thank you @JPlou for your response.

I thought about background removal but that is not enough. If I clamp everything from certain distance it leaves the ground visible (people’s bodies have to be captured in whole). Also it is not enough to clamp everything closer than where the person is cause then ground under the person is left (inside the bounding box). So I need proper segmentation and that is not possible with the sdk atm…or is there a way to get segmentation of people where arms do not get cut out?

But am I in the right path on like basically get camera image and depth data manually from ZEDCamera and do something with that data?
I know I am asking pretty basic questions but I am just figuring out optimal path to take. In my preliminary tests with python I found that the SDK’s segmentation works fine with object detection but there is the “safety feature” of cutting the arms off and body tracking is not segmenting body reliably (and also cutting body parts out) so I have been trying to find ways to get around with the limitations.

4

@tuorpete

Sorry for the delay, for the proper segmentation, we’re currently developing ways to seamlessly ingest masks when using custom object detection, which is not possible at the moment.

If you were to make segmentation on your side, you could indeed base yourself on our images, bounding boxes, and depth detection.