cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
bsarbb
Observer
Observer
670 Views
Registered: ‎03-11-2021

How to convert YUV 4:2:2 10 bpc to BGR24 ?

Jump to solution

Hello,

I am trying to convert YUV 4:2:2 10 bpc frame to BGR24 frame.

For this purpose I used AXI Subset Converter, with this IP I remove the lsb bytes of each pixel component. After this, I added "Video Processing Subsystem" to get BGR24 frame. But this configuration didn't work. 

The visualization of this explanation is : 

pipeline.png

Do you have any suggestion for this purpose ? 

Thanks.

0 Kudos
1 Solution

Accepted Solutions
bsarbb
Observer
Observer
297 Views
Registered: ‎03-11-2021

Hi @florentw

Thanks for your response,

I searched a proper way of get rid of unexpected horizontal lines and I encounter media-ctl command's "--set-v4l2" switch.

As a result, I removed the kernel patch, then when the system boot, I executed media-ctl --set-v4l2 '"80000000.v_proc_ss":0[fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]' command.

When I spesify the colorspace as rec709, color problem solved.

Here is the resulting frame:

resulting_frame1.png

I tried to show the last remaining problem in the following picture:

blank_line.png

 

I zoomed to this blank line:

zoom_blank_line.png

Do you have any idea about this problem ?

Many thanks for your time again.

View solution in original post

16 Replies
florentw
Moderator
Moderator
648 Views
Registered: ‎11-09-2015

HI @bsarbb 

First, do you know that you do not need the concat IP and the constant, you can do everything in the subset converter using the REMAP string settings in the AXI4-Stream subset converter. This is something I have used in my articles around the AXI4-Stream infrastructure IPs for Video

Video Beginner Series 12: Using the AXI4-Stream Infrastructure IP Suite (Part 1)

Video Beginner Series 13: Using the AXI4-Stream Infrastructure IP Suite (Part 2)

Then you are not explaining what you mean by it does not work

Are you getting any output but with incorrect colours or are you getting no output?

Did you try to add any ILA to investigate?


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
bsarbb
Observer
Observer
620 Views
Registered: ‎03-11-2021

Hello @florentw,

Thanks for your response, I will read your read your post immediately.

I am trying to capture frames with yavta. The actual command I used is, "yavta -c10 -f BGR24 -s 1280x720 --skip 7 -F /dev/video0".

The actual test pattern I send from SDI TX to SDI RX subsystem is something like that :

orig2.png

But the captured frame is very different from original frame. (I attached example frame "frame-000007.bin")

I can't figure out what is the problem. 

I also changed device tree and some of kernel source code to capture frame. I added pl.dtsi (auto generated from petalinux), and system-user.dtsi as an attachment.

The first thing that I've change in device tree is :

I added to reset GPIO to the FrameBuffer Write IP node in device tree. FrameBuffer Write driver probed successfully.

 

&v_frmbuf_wr_1 {
	reset-gpios = <&axi_gpio_0 0x00 0x00 0x01>; // AXI_GPIO_0, 0.pin number, first channel, ACTIVE LOW.
};

 

The second thing is :

I changed to compatible string of Video Processing Subsystem (denoted with v_proc_ss_0 label in device tree), so I forced to use "Xilinx VPSS Color Space Converter" driver. And I added endpoint for bind sdi rx side and video processing subsystem.

 

&v_proc_ss_0 {
        compatible = "xlnx,v-vpss-csc";
        reset-gpios = <&axi_gpio_0 0x00 0x08 0x01>; // AXI_GPIO_0, 0.pin number, second channel, ACTIVE LOW.
};

&csc_port0v_proc_ss_0 {
        xlnx,video-format = <0>; // See include/dt-bindings/media/xilinx-vip.h for meaning of 0.

        v_proc_ss_0v_smpte_uhdsdi_rx_ss: endpoint {
                remote-endpoint = <&sdirx_outv_smpte_uhdsdi_rx_ss>;
        };        
};

 

Third, I re-configure the output format of "Video Processing subsystem" by adding the following line to the system-user.dtsi :

 

&csc_port1v_proc_ss_0 {
        xlnx,video-format = <2>; // See include/dt-bindings/media/xilinx-vip.h for meaning of 2.
};

 

I specify that the output format of the "Video Processing Subsystem" is BGR.

And lastly, I added endpoint to sdi rx node to connect output of sdi rx subsystem to input of Video Processing Subsystem".

 

&sdirx_portv_smpte_uhdsdi_rx_ss {
        sdirx_outv_smpte_uhdsdi_rx_ss: endpoint {
                remote-endpoint = <&v_proc_ss_0v_smpte_uhdsdi_rx_ss>;
        };
};

 

 

With these changes, All drivers probed successfully and the following entries appeared in the dev directory :

 

root@hdsdi-rx-bgr:~# ls -ltr /dev | grep video
crw-rw---- 1 root video   251,   0 Jan  1  1970 media0
crw-rw---- 1 root video    81,   0 Jan  1  1970 video0
crw-rw---- 1 root video    81,   1 Jan  1  1970 v4l-subdev0
crw-rw---- 1 root video    81,   2 Jan  1  1970 v4l-subdev1

 

 

 Then I changed kernel sources a bit for capture frame with yavta tool.

First of all I disable link validation in mc-entity.c.  The reason of this change is :

The video-width of output port of sdi rx subsystem is 10 bit and input of the "Video Processing Subsystem" is 8 bit. Link validation failed because of this. So I decided to ignore link validation part.

Here is the difference:

 

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 7457f6fd4a85..00e57e7073ca 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -500,7 +500,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
                            !(link->flags & MEDIA_LNK_FL_ENABLED))
                                continue;
 
-                       ret = entity->ops->link_validate(link);
+                       ret = 0; // entity->ops->link_validate(link);
                        if (ret < 0 && ret != -ENOIOCTLCMD) {
                                dev_dbg(entity->graph_obj.mdev->dev,
                                        "link validation failed for '%s':%u -> '%s':%u, error %d\n",

 

 

and lastly, I changed the order of video formats that is defined in "xilinx-vip.c". Here is the difference:

 

diff --git a/drivers/media/platform/xilinx/xilinx-vip.c b/drivers/media/platform/xilinx/xilinx-vip.c
index 4c8603064a74..ce71a7dab3d6 100644
--- a/drivers/media/platform/xilinx/xilinx-vip.c
+++ b/drivers/media/platform/xilinx/xilinx-vip.c
@@ -40,14 +40,14 @@ static const struct xvip_video_format xvip_video_formats[] = {
          2, 12, V4L2_PIX_FMT_X016, 2, 1, 2, 2 },
        { XVIP_VF_YUV_420, 16, NULL, MEDIA_BUS_FMT_UYYVYY16_4X32,
          2, 12, V4L2_PIX_FMT_X016M, 2, 2, 1, 2 },
+       { XVIP_VF_VUY_422, 8, NULL, MEDIA_BUS_FMT_UYVY8_1X16,
+         2, 16, V4L2_PIX_FMT_UYVY, 1, 1, 2, 1 },
+       { XVIP_VF_YUV_422, 8, NULL, MEDIA_BUS_FMT_UYVY8_1X16,
+         2, 16, V4L2_PIX_FMT_YUYV, 1, 1, 2, 1 },
        { XVIP_VF_YUV_422, 8, NULL, MEDIA_BUS_FMT_UYVY8_1X16,
          1, 16, V4L2_PIX_FMT_NV16, 2, 1, 1, 1 },
        { XVIP_VF_YUV_422, 8, NULL, MEDIA_BUS_FMT_UYVY8_1X16,
          1, 16, V4L2_PIX_FMT_NV16M, 2, 2, 1, 1 },
-       { XVIP_VF_YUV_422, 8, NULL, MEDIA_BUS_FMT_UYVY8_1X16,
-         2, 16, V4L2_PIX_FMT_YUYV, 1, 1, 2, 1 },
-       { XVIP_VF_VUY_422, 8, NULL, MEDIA_BUS_FMT_UYVY8_1X16,
-         2, 16, V4L2_PIX_FMT_UYVY, 1, 1, 2, 1 },
        { XVIP_VF_YUV_422, 10, NULL, MEDIA_BUS_FMT_UYVY10_1X20,
          1, 16, V4L2_PIX_FMT_XV20, 2, 1, 2, 1 },
        { XVIP_VF_YUV_422, 10, NULL, MEDIA_BUS_FMT_UYVY10_1X20,

 

With this change, Video processing subsystem driver decided the input pixel format as "V4L2_PIX_FMT_UYVY".

 

That's all the changes I've made.

Do you notice anything wrong in my changes ? 

and Could you suggest anything by looking at the difference between original frame and captured frame ? 

 

Thanks.

0 Kudos
florentw
Moderator
Moderator
607 Views
Registered: ‎11-09-2015

HI @bsarbb 

I think we need to look at the configuration of the VPSS. If the SDI is outputting 10 bit per components then you need to configure the VPSS in 10-bits per components in Vivado.

Then probably the SDI is outputting 1 pixel per clock but the the VPSS is expecting 2 pixels per clock (if you say it is configured for 8-bit then it has to be set to 2 pixels per clock to give a datawidth of 48).

So I think you have more to look in the Vivado design than in the device tree + linux application. The changes in the DT does make sense to me (reading quickly).


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
bsarbb
Observer
Observer
498 Views
Registered: ‎03-11-2021

Hello @florentw,

Actually there are not many configuration option in Vivado side, so I will share the configurations as png. Maybe you may notice something wrong.

The vpss configuration is "vpss_config.png" and frame buffer write configuration is "frmbuf_write_config.png"

Thanks a lot for your time.

vpss_config.png
frmbuf_write_config.png
0 Kudos
florentw
Moderator
Moderator
477 Views
Registered: ‎11-09-2015

Hi @bsarbb 

SO the SDI IP is outputting 10 bits per components but you are setting the VPSS IP to 8 bits per components? Is it expected in your design?

If yes, have you thought about how the data should be connected to do 10 bits -> 8 bits . Can you share what remapping you did in the AXI4-Stream subset converter?

 


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
bsarbb
Observer
Observer
466 Views
Registered: ‎03-11-2021

Hello @florentw,

SDI outputting YUV4:2:2 10 bpc and then we do 10 bit -> 8 bit conversion with AXI Subset Converter. Here is the configuration screen of AXI Subset Converter: subset_conv_config.png

We follow the UG934's "Encoding Multiple Pixels - Static TDATA Configuration" section so We ignore the LSB 2 bit of each component.

btw, I follow your suggestion and I removed concat IP from our design. As a result, the final design becomes like : 

final_design.png

0 Kudos
florentw
Moderator
Moderator
457 Views
Registered: ‎11-09-2015

Ok. That looks good.

Could you generate the image output in a common format like JPEG so I can have a quick look?


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
bsarbb
Observer
Observer
414 Views
Registered: ‎03-11-2021

Hello @florentw,

Thanks for your response.

Here is one resulting frame. I also zoom (x4, x16) to resulting picture and added as a picture here.

result_screenshot1.png

x4 zoom:

result_screenshot_2.png

x16 zoom:

result_screenshot_3.png

Many thanks for your time. 

0 Kudos
florentw
Moderator
Moderator
385 Views
Registered: ‎11-09-2015

HI @bsarbb 

Thanks. So this is not too far but there is a mismatch of the data.

Could you send me your final DT?

What you can eventually do as well is add an ILA before and after the VPSS. This way you can see if it is correctly doing the CSC or not


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
bsarbb
Observer
Observer
375 Views
Registered: ‎03-11-2021

Hi @florentw,

I will quickly add ILA and then evaluate results and share with you.


Also,
I recorded resulting device tree with "dtc -I fs /sys/firmware/devicetree/base > dev_tree.txt 2>&1" command. I added this "dev_tree.txt" as an attachment.

Thank you.

0 Kudos
bsarbb
Observer
Observer
350 Views
Registered: ‎03-11-2021

Hi @florentw

I checked the kernel driver of "Video Processing Subsystem" and I realize that the input and output video formats set in the probe function with "xcsc_set_default_state" function call.

static void xcsc_set_default_state(struct xcsc_dev *xcsc)
{
	xcsc->cft_in = XVIDC_CSF_RGB; 
	xcsc->cft_out = XVIDC_CSF_RGB;
	xcsc->output_range = XVIDC_CR_0_255;
	/* Needed to add 10, 12 and 16 bit color depth support */
	xcsc->clip_max = BIT(xcsc->color_depth) - 1;
	xcsc_set_control_defaults(xcsc);
	xcsc_set_unity_matrix(xcsc);
	xcsc_write(xcsc, XV_CSC_INVIDEOFORMAT, xcsc->cft_in);
	xcsc_write(xcsc, XV_CSC_OUTVIDEOFORMAT, xcsc->cft_out);
	xcsc_write_coeff(xcsc);
	xcsc_write(xcsc, XV_CSC_CLIPMAX, xcsc->clip_max);
	xcsc_write(xcsc, XV_CSC_CLAMPMIN, XCSC_CLAMP_MIN_ZERO);
}

These input and output video format values don't change when I try to get BGR24 frames.

I executed "yavta -c10 -f BGR24 -s 1280x720 --skip 7 -F /dev/video0" command but before this I enabled the debug messages in "xcsc_s_stream" function. (defined in xilinx-vpss-csc.c).

When I executed above yavta command, I saw following debug messages in dmesg:

[   47.348907] xilinx-vpss-csc 80000000.v_proc_ss: xcsc_s_stream : Stream On
[   47.361211] xilinx-vpss-csc 80000000.v_proc_ss: -------------CSC Coeff Dump Start------
[   47.374743] xilinx-vpss-csc 80000000.v_proc_ss:  R row :  4096      0      0
[   47.387321] xilinx-vpss-csc 80000000.v_proc_ss:  G row :     0   4096      0
[   47.399906] xilinx-vpss-csc 80000000.v_proc_ss:  B row :     0      0   4096
[   47.412489] xilinx-vpss-csc 80000000.v_proc_ss: Offset :     0      0      0
[   47.425077] xilinx-vpss-csc 80000000.v_proc_ss: ClampMin:   0  ClipMax 255
[   47.437486] xilinx-vpss-csc 80000000.v_proc_ss: -------------CSC Coeff Dump Stop-------
[   47.451038] xilinx-vpss-csc 80000000.v_proc_ss: cft_in = 0 cft_out = 0
[   47.463100] xilinx-vpss-csc 80000000.v_proc_ss: clipmax = 255 clampmin = 0
[   47.475517] xilinx-vpss-csc 80000000.v_proc_ss: height = 720 width = 1280
[   47.720482] xilinx-vpss-csc 80000000.v_proc_ss: xcsc_s_stream : Stream Off

 

From these output messages, I decided to change input format of "Video Processing Subsystem" by hand. So, here is the change:

static void xcsc_set_default_state(struct xcsc_dev *xcsc)
{
	xcsc->cft_in = XVIDC_CSF_YCRCB_422; //XVIDC_CSF_RGB;
	xcsc->cft_out = XVIDC_CSF_RGB;
	xcsc->output_range = XVIDC_CR_0_255;
	/* Needed to add 10, 12 and 16 bit color depth support */
	xcsc->clip_max = BIT(xcsc->color_depth) - 1;
	xcsc_set_control_defaults(xcsc);
	xcsc_set_unity_matrix(xcsc);
	xcsc_write(xcsc, XV_CSC_INVIDEOFORMAT, xcsc->cft_in);
	xcsc_write(xcsc, XV_CSC_OUTVIDEOFORMAT, xcsc->cft_out);
	xcsc_write_coeff(xcsc);
	xcsc_write(xcsc, XV_CSC_CLIPMAX, xcsc->clip_max);
	xcsc_write(xcsc, XV_CSC_CLAMPMIN, XCSC_CLAMP_MIN_ZERO);
}

 

After this change I get the following frame:

resulting_frame1.png

x4 zoom:

resulting_frame2.png

So, The unexpected horizontal lines disappeared after this change. But the color is still wrong.

Is this change the most proper way to get rid of unexpected horizontal lines or is there a better way  ? And lastly, Could the cause of the color error be similar ? 

Thanks again for your time. 

0 Kudos
florentw
Moderator
Moderator
329 Views
Registered: ‎11-09-2015

Hi @bsarbb 

Can you share the debug output after editing the driver? I would like to see what coefficients are used.


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
bsarbb
Observer
Observer
298 Views
Registered: ‎03-11-2021

Hi @florentw

Thanks for your response,

I searched a proper way of get rid of unexpected horizontal lines and I encounter media-ctl command's "--set-v4l2" switch.

As a result, I removed the kernel patch, then when the system boot, I executed media-ctl --set-v4l2 '"80000000.v_proc_ss":0[fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]' command.

When I spesify the colorspace as rec709, color problem solved.

Here is the resulting frame:

resulting_frame1.png

I tried to show the last remaining problem in the following picture:

blank_line.png

 

I zoomed to this blank line:

zoom_blank_line.png

Do you have any idea about this problem ?

Many thanks for your time again.

View solution in original post

florentw
Moderator
Moderator
263 Views
Registered: ‎11-09-2015

HI @bsarbb 

Good to know that you have made progress on this.

No I have no idea from where the line can be coming. But we can see that when the VPSS was not doing any CSC the line was here.

So I suggest you add an ILA and try to check the values of the first pixels of the line to see if this is coming from SDI or later.


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
bsarbb
Observer
Observer
169 Views
Registered: ‎03-11-2021

Hi @florentw,

I added ILA to the SDI-RX subsystem output as you suggested and I find out that the source of the problem is SDI-TX side. Our digital design engineers finds the bug and issue is solved.

Thanks for your help.

florentw
Moderator
Moderator
157 Views
Registered: ‎11-09-2015

Hi @bsarbb 

Great. Good to read that you have found the issue. Could you kindly close the thread by marking a reply as accepted solution? I recommend marking your previous reply which contained the solution for the initial issue.

Regards


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos