UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Contributor
Contributor
2,017 Views
Registered: ‎09-19-2017

What's the output format of omxh265dec/omxh264dec on zcu106 board

Jump to solution

Hi,

 

I'm trying to make vcu decode rtsp streams and feed frames to customized CNN accelerator. Here is my gstreamer pipeline configuration for decoding h265 stream:

 

rtspsrc location=<uri> ! rtph265depay ! h265parse ! omxh265dec ip-mode=1 op-mode=1 ! videoconvert ! appsink name=sink caps="video/x-raw, format=BGR"
 
It works correct but extremely slow (>500ms per frame) on zcu106 board because the videoconvert element takes to much time. Then I tried to use opencv to replace the videoconvert element. Here is the new pipeline configuration:
 
rtspsrc location=<uri> ! rtph265depay ! h265parse ! omxh265dec ip-mode=1 op-mode=1 ! appsink name=sink
 
I use cv::Mat yuv = cv::Mat(int(height)*1.5, int(width), CV_8UC1, (void *)map.data) to initiate the cv::Mat object on GstMapInfo object's internal pointer which is gathered from pull-sample operation on the appsink element. Then I use cv::cvtColor(yuv, img, CV_YUV2BGR_NV12) function to convert the color. The result is completely incorrect: when decoding some rtsp sources, it displays gray color with green region on the top part of the image; the other sources look completely incorrect (hard to describe it. It feels more like a completely distorted image, but can see the screen turns black when use my hand to cover the camera).
 
What's the possible cause of this? From the VCU document, it says it supports outputting both yuv422 and yuv420 color in either 8-bit or 10-bit format. How can I configure the omxh265dec to output the specified color format? How can I properly convert the frame format to BGR? Can xfopencv also works?
 
I also found that the GstBuffer object contains 3342336 byte (1080p source) of data using gst_buffer_get_size function call. If it is a standard YUV420sp, the size should be 3110400 bytes. I have tested by program on PC using PC's codec - avdec_h265. All works fine, the buffer size is 3110400, the opencv program can convert the raw YUV_I420 to BGR and imshow it correctly.
 
I also tried h264 rtsp source. On some sources it can work but just for a couple of frames (between two i frames) and then not responding anymore. Is it because the missing of SPS and PPS on the second i frame?
 
Reimond
 
 
0 Kudos
1 Solution

Accepted Solutions
Contributor
Contributor
2,590 Views
Registered: ‎09-19-2017

Re: What's the output format of omxh265dec/omxh264dec on zcu106 board

Jump to solution

I finally found the point! When I set the omxh265dec op-mode to 0, the returned GstBuffer stores the NV12 data in correct format. The cv::cvtColor now can correctly convert the color to BGR. Here is the complete configuration:

 

rtspsrc location=<rtsp uri>  ! rtph265depay ! h265parse ! omxh265dec ip-mode=1 op-mode=0 ! appsink name=sink

 

When op-mode is set to 1, the returned GstBuffer is DMA buffer. I'm not sure how the data is organized in this container, but it cannot be used directly.

 

But h264 streams still have problems. The decoder can only decode the first section of frames before the second i frame. Not sure how to solve it. My colleague suggests me to add SPS and PPS to every i frame, but it seems to be a big workload now.

3 Replies
Xilinx Employee
Xilinx Employee
1,951 Views
Registered: ‎08-01-2007

Re: What's the output format of omxh265dec/omxh264dec on zcu106 board

Jump to solution

You will want to make sure that you are using NV12 format.

 

I recommend you start by looking at the Debug Section of PG252.  It has some sample GStreamer pipelines in the "Debugging Gstreamer Based Application" section.

Chris
Video Design Hub | Embedded SW Support

---------------------------------------------------------------------------
Don’t forget to Reply, Kudo, and Accept as Solution.
---------------------------------------------------------------------------
Contributor
Contributor
1,942 Views
Registered: ‎09-19-2017

Re: What's the output format of omxh265dec/omxh264dec on zcu106 board

Jump to solution

Thanks for the reply.

 

I customized the trd design of zcu106 board: all of the hdmi rx-tx logic is removed from the project in order to leave more space for my sdsoc accelerator. The vcu part is identical to the original trd design. The platform can run the qt demo but only limited to play the 4k video file.

 

The pipeline configuration:

 

rtspsrc location=<rtsp uri> ! rtph265depay ! h265parse ! omxh265dec ip-mode=1 op-mode=1 ! videoconvert ! appsink name=sink caps="video/x-raw,format=BGR"

 

can work but just videoconvert element converting omxh265dec's output to BGR format takes too much time and hope to use pl accelerator to replace it. I then changed the pipeline to 

 

rtspsrc location=<rtsp uri>  ! rtph265depay ! h265parse ! omxh265dec ip-mode=1 op-mode=1 ! appsink name=sink

 

I use gchar *format = gst_structure_get_string (structure, "format"); to confirm that the omxh265dec is outputting NV12 frames, but the returned buffer size looks strange: it should be 1920x1080x1.5=3,110,400, but actually it is reporting 3,342,336 bytes. I use gsize buffer_size = gst_buffer_get_size(buffer); to get the buffer size. 

 

All I want to know is the actual layout of the returned buffer which part is Y channel, and which part is the UV channel so that I can write an accelerator to transfer it to BGR format.

0 Kudos
Contributor
Contributor
2,591 Views
Registered: ‎09-19-2017

Re: What's the output format of omxh265dec/omxh264dec on zcu106 board

Jump to solution

I finally found the point! When I set the omxh265dec op-mode to 0, the returned GstBuffer stores the NV12 data in correct format. The cv::cvtColor now can correctly convert the color to BGR. Here is the complete configuration:

 

rtspsrc location=<rtsp uri>  ! rtph265depay ! h265parse ! omxh265dec ip-mode=1 op-mode=0 ! appsink name=sink

 

When op-mode is set to 1, the returned GstBuffer is DMA buffer. I'm not sure how the data is organized in this container, but it cannot be used directly.

 

But h264 streams still have problems. The decoder can only decode the first section of frames before the second i frame. Not sure how to solve it. My colleague suggests me to add SPS and PPS to every i frame, but it seems to be a big workload now.