I see is possible to use chorma keying with the camera like the demo shows. But I want to use passthrough with the green screen, so whatever is green screen is virtual, and the rest is just the camera. How can I do that?
Hi @NicolasAbo17, welcome to the forums.
This is an interesting project!
Have you tried piecing together the AR Passthrough example and the Green Screen example?
We do not provide a guide for this (haven’t tried it myself), but I’d be happy to help you along the way.
Do not hesitate if you have questions during the process.
Best,
Jean-Loup
In summary, I did the tutorials and the problem is the Green Screen prefab uses the Rig_mono as base, but I need to use the Rig_stereo because I want the chroma keying in the headset, not in a static camera. There is no clear guide on how to apply the green screen script properly to the rig_stereo and in the documentation it says that to add a VR headset you just have to add the green screen prefab and the steam vr camera rig prefab, but that doesn’t work either. Do you have a developer or who I can contact to get help for this? I know the technology is capable of doing what I want but I can’t find help anywhere. This is an example of what I want: ZED Mini Greenscreen - Mixed Reality - YouTube
@NicolasAbo17 I’m the dev you’re looking for 
Unfortunately, it was very difficult to take the time for this question this week, as it requires diving a bit into the code.
I encourage you to try things out, as it’s not a use case that was planned when the samples were created (and it’s not a part that was kept up-to-date that well, as the green screen use case is largely not the main one for the plugin 
). That being said, I think that the code has some useful comments in it, and the issues probably lay in how to set the right textures/material on the stereo eyes, if you find a way to address this, feel free to open a pull request on the repo. While it’s a particular use case, it is an interesting one.
I’ll try to take a more in-depth look as soon as I can if it’s not resolved by then.
Thanks for your understanding.
@JPlou
I think I solved it, in how to apply the green screen manager to a rig stereo. I added the green screen manager to each eye and the material to each frame, but I had to add code to the green manager script, because for some reason the screenManager.matRGB was null:
"
private void ZEDReady()
{
…
//We set the material again in case it has changed.
//screen = gameObject.transform.GetChild(0).gameObject;
if (screenManager.matRGB == null)
{
screenManager.SetMatRGB();
}
finalMat = screenManager.matRGB;
"
It works as the passthrough demo, but with the green screen effect, I also created a simple green screen manager for both green screen manager, just to change things like color, range, smoothness, etc. in the same script.
But the problem that I now have is that 3D objects are always prioritized. The chroma key seems to only apply to the skybox, and the rest works with the depth. Is there a way, even for the normal ZEDGreenScreen prefab, to make that only virtual objects appear in the real world green screen? I have this issue even when testing with the ZEDGreenScreen prefab, so if I make it work for that one it probably will work in the ZED_Rig_Stereo.
Someone suggested me that maybe I could get the render texture of the normal passthrough camera and apply a different green screen shader that the one in the zed plugin. But I have no idea how to do that or if is possible. Or if you now of a different solution that could help me.
1 Like
@JPlou Do you have a suggestion? I am trying to change the shader but I’m not getting the desired effect, the virtual objects when are not in the green screen area, they work with depth, but I want them to disappear always if they are not part of the green screen area. And when I try third party shaders it always changes blue and red colors and looks weird, I don’t know what to do.
@NicolasAbo17
Yeah, the plugin was made like this intentionally. But don’t worry, it’s definitely possible since it was done
I played around a bit, and found a solution, but it’s maybe a bit much for VR, since it implies running 4 cameras for stereo. It could also work perfectly, I don’t know
I did it on the mono sample:
- Create some dedicated layers, and put all your 3D environment inside
- Create a camera along the Camera_Left (I just copied camera_Left and scrapped the ZED stuff)
- Set this camera:
- culling mask to only your 3D scene layers
- Clear Flags to Skybox
- Depth to 1
- Change some Camera_Left settings:
- Clear Flags to Depth Only
- Culling Mask to the same it was, don’t put your 3D scene layers here
- Depth to something greater than 1
And yeah, that’s it 
Let me know if it works well in VR!
Thanks but is not working, when I look at the green screen, real life items get duplicated. This happens in unity editor, but when I put on the headset the left eye gets stuck in the Unity splash screen, and the right eye just shows the virtual environment but does not respond to movement or anything, is just static. 4 cameras probably doesn’t work with how the Rig_Stereo is constructed. I found a shader that does what I want, but is inverting real life red and blue colors, maybe the solution is in finding the correct shader or fix it, but I don’t know anything about shaders. Still this shader that I found is not as good as the one the ZED plugin has
If there are some overlapping issues, maybe you have to play with the “depth” setting of the Camera (not a ZED parameter of any case, the order of rendering things in the scene). I had issues when I put identical depths for example.
In your shader, you probably just have to invert two components of a vector at some point, you should find it out quickly by trying things (something the r and b values of a “color” vector in the shader, by what you’re saying).
The main problem is more the way 4 cameras don’t work with the Rig_stereo. Even if I solved the issue of the duplicates it doesn’t seem to work. And that shader I don’t know is weird, I don’t see r and b values, it only takes the bgr color, I changed it to rgb but it didn’t do anything.