How to handle segmentation mask with OpenCVSharp?

Hello,

I am looking for the appropriate way to handle the body tracking segmentation mask received from BodyData.mask.

I am trying to get an image that only shows the first detected body, based on the segmentation mask of the body tracking module.
For image processing I am using OpenCVSharp to make an OpenCvSharp.Mat where data is the IntPtr contained in BodyData.mask. I am able to create this matrix but a System.AccessViolationException appends when I want to clone it with the Mat.Clone() method to manipulate a copy of it.

I join you an example to reproduce my exception based on the C# implementation of the body tracking sample where I have activated enableSegmentation in the bodytracking parameters and tried to access to mask in the grabbing frame loop:

    // Render loop
    private void NativeWindow_Render(object sender, NativeWindowEventArgs e)
    {
        OpenGL.CoreUI.NativeWindow nativeWindow = (OpenGL.CoreUI.NativeWindow)sender;
        Gl.Viewport(0, 0, (int)nativeWindow.Width, (int)nativeWindow.Height);
        Gl.Clear(ClearBufferMask.ColorBufferBit);

        if (viewer.isAvailable() && zedCamera.Grab(ref runtimeParameters) == ERROR_CODE.SUCCESS)
        {
            if (imageLeft.IsInit())
            {
                // Retrieve left image
                zedCamera.RetrieveMeasure(pointCloud, sl.MEASURE.XYZRGBA, sl.MEM.CPU, pcRes);
                zedCamera.RetrieveImage(imageLeft, sl.VIEW.LEFT, sl.MEM.CPU, displayRes);
                zedCamera.GetPosition(ref camPose, REFERENCE_FRAME.WORLD);

                // Retrieve Objects
                zedCamera.RetrieveBodies(ref bodies, ref bt_runtime_parameters);

                TrackingViewer.render_2D(ref imageLeftOcv, imgScale, ref bodies, isTrackingON, bt_params.bodyFormat);

                //Update GL View
                viewer.update(pointCloud, bodies, camPose);
                viewer.render();

                if (isPlayback && zedCamera.GetSVOPosition() == zedCamera.GetSVONumberOfFrames()) return;

                foreach (var body in bodies.bodiesList)
                {
                    if (body.trackingState == OBJECT_TRACKING_STATE.OK)
                    {
                        var bb = body.boundingBox2D;

                        var topLeftCorner = bb[0];
                        var topRightCorner = bb[1];
                        var bottomRightCorner = bb[2];
                        var bottomLeftCorner = bb[3];

                        var width = (int)Math.Abs(topRightCorner.X - topLeftCorner.X);
                        var height = (int)Math.Abs(bottomLeftCorner.Y - topLeftCorner.Y);

                        using (var maskOcv = new OpenCvSharp.Mat(height, width, MatType.CV_8UC4, body.mask))
                        {
                            imageLeftOcv = maskOcv.Clone();
                        }
                        break;
                    }
                }
                Cv2.ImShow(window_name, imageLeftOcv);
            }
        }
    }

The error message:
Exception non gérée : System.AccessViolationException: Tentative de lecture ou d’écriture de mémoire protégée. Cela indique souvent qu’une autre mémoire est endommagée.
Ă  OpenCvSharp.Internal.NativeMethods.core_Mat_clone(IntPtr self, IntPtr& returnValue)
Ă  OpenCvSharp.Mat.Clone()
Ă  sl.MainWindow.NativeWindow_Render(Object sender, NativeWindowEventArgs e) dans MainWindows.cs:ligne 267

“Thank you in advance for your assistance!”

Hi @VR_Neo, welcome to the forum!

First of all, I think you’re retrieving the mask in the wrong type, it should be CV_8UC1 since it’s an array of int from 0 to 255, one value per pixel, you will probably have weird masks if it worked as is.
I’ve not managed to get a clean mask 100% of the time on my side either, I’ll take a deeper look tomorrow, sorry for the delay.

As an alternative to the mask, you could use the bounding box to create a window just around the person you’re tracking, would that work for you?

Hello @JPlou,

Yes I didn’t think that the mask is a two dimensions matrix so it makes more sense to use CV_8UC1 . I changed my code with this fix but it’s not the main reason of this exception…

Using bounding box is not enougth in my case, I need a clean segmentation for Mixed Reality purpose. I suppose that I could use another segmentation neural network on the content of the bounding box but I’m afraid that adding this step won’t be effective in real time.

Hi,

I found some issues in the csharp wrapper preventing the retrieve of the mask.
I’m fixing it and it will be available in the next ZED SDK release (4.1).

I also updated the Object Detection sample to correctly display the masks on top of the image.

Best,
Benjamin Vallon

1 Like