OpenCV Videowriter not able to write Zed stream to RTSP server

joev · November 20, 2022, 3:15am

I am using a cv2.Videowriter to try and write the Left Camera stream to an RTSP server using python. But It dose not seem to be able to work.

Now I am able to view the left stream from the camera with

cv2.imshow(‘Image’, image_ocv)

But when I use

out_send.write(image_ocv)

to write to the RTSP server I get this error

WARN:0] global /home/joev/JEP/script/workspace/opencv-4.5.0/modules/videoio/src/cap_gstreamer.cpp (1631) writeFrame OpenCV | GStreamer warning: cvWriteFrame() needs images with depth = IPL_DEPTH_8U and nChannels = 3.

Here is my out send string

out_send = cv2.VideoWriter(‘appsrc ! videoconvert ! video/x-raw,format=YUY2 ! nvvidconv ! video/x-raw(memory:NVMM),format=(string)NV12,width=1280,height=720 !
nvv4l2h264enc bitrate=12000000 insert-sps-pps=1 idrinterval=1 insert-vui=1 ! video/x-h264
! rtph264pay pt=96 !
udpsink host=127.0.0.1 port=5400 async=false sync=0’,
cv2.CAP_GSTREAMER, 0, 30, (1280,720), True)

and here is my RTSP factory set launch string

factory.set_launch(“(udpsrc name=pay0 port=%d buffer-size=524288 caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=(string)%s, payload=96 " )” % (updsink_port_num, codec))

I have tried numerous combinations for the
out_send = cv2.VideoWriter string
but nothing seems to get past the error

alassagne · November 21, 2022, 12:23pm

Hello,

It seems that your image_ocv has the wrong format. Your missing channels, or you have too many channels. Try to convert your matrix to the right format first.

Antoine

joev · November 21, 2022, 1:48pm

I agree with you conclusion. Now the trick is to find out what type of matrix the “out.write”
wants to see.
Im curious how cv2.imshow is able to parse the same matrix and show the stream from the zed camera.

Does anybody out there know how the “get data” converts to a NumPy array
to be used with opencv?

alassagne · November 21, 2022, 1:55pm

The get_data actually does not copy anything, it’s a shared pointer. You can check what it does on github, our wrapper is open source.

Antoine

joev · November 21, 2022, 4:10pm

I searched in the github repo for “get_data:” from the wrapper to see what it does but was not able find it.
I found code examples using “get_data:” .

Would you be able to point me in a direction in the github repo to find how “get_data:” works
within the wrapper.

joev · November 21, 2022, 7:41pm

To recover data from sl.Mat to use it with opencv, use the get_data() method
# It returns a numpy array that can be used as a matrix with opencv
image_ocv = image_zed.get_data()

Does image_zed.get data() grab a specidic location from sl.Mat?

alassagne · November 22, 2022, 9:02am

Hello, the function is here :

github.com

stereolabs/zed-python-api/blob/master/src/pyzed/sl.pyx#L3734

    
      
          # Returns the format of the matrix.
          # \return The format of the current \ref Mat
          def get_memory_type(self):
              return MEM(<int>self.mat.getMemoryType())
          
          
##
          # Cast the data of the \ref Mat in a Numpy array (with or without copy).
          # \param memory_type : defines which memory should be read. Default: [MEM.CPU](\ref MEM) (you cannot change the default value)
          # \param deep_copy : defines if the memory of the Mat need to be duplicated or not. The fastest is deep_copy at False but the sl::Mat memory must not be released to use the numpy array.
          # \return A Numpy array containing the \ref Mat data.
          def get_data(self, memory_type=MEM.CPU, deep_copy=False):
              
              shape = None
              cdef np.npy_intp cython_shape[3]
              cython_shape[0] = <np.npy_intp> self.mat.getHeight()
              cython_shape[1] = <np.npy_intp> self.mat.getWidth()
              cython_shape[2] = <np.npy_intp> self.mat.getChannels()
          
          
    if self.mat.getChannels() == 1:
                  shape = (self.mat.getHeight(), self.mat.getWidth())
              else:

You should find everything you need.

Antoine

joev · November 22, 2022, 10:44am

Awesome
Thanks
Just a bit of info on what I am trying to accomplish.

I have modified the python github ZED-Opencv example so it can retrieve the Zed camera stream. Then get the depth data from any pixel that is clicked on.
Then I am attempting to send the stream back out RTSP to a Nvidia Deepstream YOLO pipeline for object detection.

alassagne · November 22, 2022, 10:58am

You can use our streaming sample, it uses RTP. you can then do your object detection on your sl.Mat or CV.Mat. You won’t need to convert the sl.Mat to opencv much later.

Antoine

joev · November 24, 2022, 12:57pm

So the issue was

out_send.write

wants to see a RGB stream and

image_ocv = image_zed.getdata()

returns a RGBA stream

So the solution is

image_RGB = cv2.cvtColor(image_ocx, cv2.COLOR_RGBA2RGB)
out_send.write(image_RGB)

Again thanks for the help.
Once I get it all cleaned up I will post a video with explanation and code.

alassagne · November 24, 2022, 1:32pm

Glad you could find a solution