Build Information:
-Ubuntu 18.04.6 LTS
-Nvidia Jetson Xavier NX
-Jetpack4.6
-ZED2i Camera
-ZED SDK 3.6.3 (for Xavier NX)
Problem 1: I am having problems getting the depth camera stream to display correctly when using zedsrc stream-type:4 (Left_and_Depth_up/down [BGRA] - 8 bits- 4 channels Left and Depth(image)). When running the pipeline, I am receiving a mostly black output with some white streaks which are roughly the hardest/strongest edges in the image (ex: an open door frame). Running the ZED Depth Viewer tool from the SDK on the same system, I get the expected depth image, but not through gstreamer. How can I get the output for the depth stream to display properly through a gstreamer pipeline?
For reference: I am running the following pipeline via the following command line input:
gst-launch-1.0 zedsrc stream-type=4 ! zeddemux name=demux \
demux.src_left ! queue ! autovideoconvert ! ximagesink \
demux.src_aux ! queue ! autovideoconvert ! ximagesink
(derived from GStreamer - ZED Demux | Stereolabs)
Problem 2: The goal of the above pipeline was to eventually convert it into c++ code. I have successfully written the c++ app that displays both the left image and the depth map (with the same problems as Problem 1). I was wondering if this was a conversion issue from the sink element so I added a GStreamer probe to the zeddemux’s src_aux pad. (Below, is the gstreamer probe I have added to said pad to read the frame data.) However, the buffer, when read, it only has either a 0 or 1 (probably reflecting the same issue as Problem 1 above which is why I asked this question secondly). I have tried the same probe with zeddemux’s src_left pad and successfully gotten the RGBA values. Thus, my question is:
If I am doing this correctly, how can I obtain a readable depth map from GStreamer? The goal would eventually to be to publish depth data to other applications for specific pixel values.
Probe Function:
static GstPadProbeReturn depth_buffer_probe(GstPad* pad, GstPadProbeInfo* info, gpointer u_data){
// meta data
GstBuffer *buf = (GstBuffer*) info->data;
// peeking memory block 0 containing image data
GstMemory *mem = gst_buffer_peek_memory(buf, 0);
// getting size of memory block
long int size = gst_memory_get_sizes(mem, NULL, NULL);
/*
getting an image frame that contains the depth map and looking at the
depth values around the center of the screen
*/
GstMapInfo inf;
if(gst_memory_map(mem, &inf, GST_MAP_WRITE)){
guint16 * ptr = (guint16 *) inf.data; // pointer to the image data
int width = 1920; // width of the camera image
int height = 1080; // height of the camera image
int range = 10; // range of pixels around the preceeding and following the center screen pixel
int center = (height / 2) * width; // center of the image
int start = center - range;
int end = (center + range);
/*
checking the depth value of the pixels around the center of the screen
within the range [center - range, center + range]
*/
while(start < end){
guint16 depth; // depth value at a given pixel
temp = ptr[start];
// getting the location of the pixel being looked at and printing this information
int row = (start / width) % height;
int col = (start % width);
std::cout << "row:" << row << ", col: " << col;
// printing the hexadecimal value of the depth at a given pixel
printf(", val: %04x\n", depth);
start++;
}
gst_memory_unmap(mem, &inf);
}
return GST_PAD_PROBE_OK;
}
The above probe function is set as a callback function with the following code line:
gst_pad_add_probe (zeddemux_srcpad_aux, (GstPadProbeType) \ GST_PAD_PROBE_TYPE_BUFFER), depth_buffer_probe, NULL, NULL);