cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted
1,879 Views
Registered: ‎05-29-2019

VID_Phy/Hdmi_rx_ss drivers run reliability issue

Hello,

We are using Zynq xczu4ev-fbvb900-1-e chip in our HDMI RX system solution.

There are 2 RX HDMI channels in our Zynq appliance. 

The Xilinx code (https://github.com/Xilinx/embeddedsw/tree/master/XilinxProcessorIPLib/drivers) for VID_Phy/Hdmi_rx_ss is running in Linux user space on APU-PS.

We are experienceing an issues during realibility test.

The test: HDMI source switches to different video formats periodically (10 sec).

The software compares the source format with the one discovered after RX stream is up (like in the example below the format matchs).

INFO : [RxStreamUpCallback1:1328] RX stream is up
INFO : Mode: HDMI
Color Format: YUV_422
Color Depth: 12
Pixels Per Clock: 2
Mode: Progressive
Frame Rate: 60Hz
Resolution: 1920x1080@60Hz


In the case it doesn't match we log an error.

In most of cases when input format was changed - HDMI driver wasn't capable to recognize the change and internal driver state machine was stopped in the middle, waiting for something.

When signal will be changed next time with high probability drivers will detect it correctly.

We tried a different modes of operation, but error rate is still high.

Even running only single RX channel (closing all other sw processes) we got 2 errors after 500 iterations of the test.

When running the whole user application, including 2 RX HDMI the error rate close to 10% !!!

I'm wondering if running RX HDMI drivers on some "bare metal" processor could help on this issue.

Please urgent help.  

Best Regards.

Boris Barak.

Tags (2)
0 Kudos
54 Replies
Highlighted
Xilinx Employee
Xilinx Employee
877 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

I have generated bare-metal design, followed your test sequence, and tested out the mentioned pattern, I notice there is a problem with 1920x1080i50 format. I can see "Vic and video timing mismatch" I will file a change request to report this issue.

Can you remove this format from your test sequence, and then see if other color format has such problem or not? Attached is terminal log.

Please note HDMI Rx doesn't report 59.94Hz, but you can judge that based on RX reference clock frequency. When it's 148342784 Hz, it means the frame rate is 59.94. When it's around 148.5 Mhz, the frame rate is 60/50 Hz

0 Kudos
Highlighted
815 Views
Registered: ‎05-29-2019

 

Hello,

The device is in QA now and they come back with the issue.

Here is QA scenario to reproduce it:

While channel is playing, change source formats every 30 seconds:

{1080p59.94 --> 1080p23.976 --> 1080p25 --> 1080p23.976->Repeat this cycle}

Do you have any new workaround idea to resolve this issue ?

As was mentioned in my previous posts, in the case of such kind of detection error the system is not bounded and has a difficulties to converge.

 

Best Regards.

Boris Barak.

 

0 Kudos
Highlighted
811 Views
Registered: ‎05-29-2019

I've attached again the plot of timing fluctuation in the case of the problem.

"cnt" represents a number of attempts of driver to converge.

In such case hdmi input became non responsible at all.

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
742 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

I've confirmed "Vic and video timing mismatch" issue will be fixed in the next release of Vivado.

I'm a bit confused about timing fluctuation problem. Is it related to the previous link quality issues? 

0 Kudos
Highlighted
737 Views
Registered: ‎05-29-2019

Hi,

Yes, the system is still suffering from timing fluctuation problem from time to time.

It's most likely happen in 1080p low frame rates formats (1080p24, 1080p23.976, 1080p25).

Please note that system simply stuck in this case and doesn't not recovered, until HDMI cable re-connection or new format initiated by the source.

So the severity of this issue is critical and I have no idea how to mitigate it.

Please note that this design is base for the series of our products, so it's very important to understand and resolve this.

Best Regards.

Boris Barak.

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
734 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

I will do some tests with 1080p24, 1080p23.976 (need to check if I can find source for this), 1080p25.

When timing fluctuation problem occurs, you use our example menu to type "i" and "z", what do you get?

0 Kudos
Highlighted
721 Views
Registered: ‎05-29-2019

Hi,

As I've mentioned we don't have Xilinix evaluation board and don't running example design. 

What the operation of example menu, typing "i" and "z" actually does ?

Please note that our system design is mature enough and behavior well.

The Rx detection error rate for all formats is very low.

On the other hand the error rate for the formats I've mentioned in previous post is above 4%.

It's also happen with different HDMI sources.

Boris Barak.

 

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
718 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

'i' menu calls

XV_HdmiRxSs_ReportInfo(&HdmiRxSs);

XVphy_ClkDetGetRefClkFreqHz(&Vphy, XVPHY_DIR_RX));

XVphy_DruGetRefClkFreqHz(&Vphy))

XVphy_HdmiDebugInfo(&Vphy, 0, XVPHY_CHANNEL_ID_CH1);

'z' menu calls

XVphy_LogDisplay(&Vphy);

XV_HdmiRxSs_LogDisplay(&HdmiRxSs);

If I can't figure out the problem, I probably will send you the HDMI Rx with debug port to root cause the issue.

With 'i' and 'z' menu, we can know the issue is inside Video PHY or HDMI Rx.

Highlighted
712 Views
Registered: ‎05-29-2019

We've already using some of the report functions you've mentioned, in our test.

It makes sense to call all these functions in the case of error happens, indeed.

I'll continue an investigation.

In the case you we'll have some results on your side please let me know. 

Thank You.

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
696 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

The attachment only has video timing. If it's not in the supported table, have you followed https://www.xilinx.com/support/answers/68227.html to add custom resolution?

HDMI Rx driver doesn't support/report non-integer frame rate, but it can receive it if the clock frequency is correct. The way to judge it is to read out the clock freq from Video PHY register or use the API to print out the frequency.

 

Highlighted
674 Views
Registered: ‎05-29-2019

Hi,

For the time being we don't need any custom resolution, that not already implemented by Xilinix.

The following resolutions are supported, I've re-checked in the driver code.

XVIDC_VM_1920x1080_24_P,
XVIDC_VM_1920x1080_25_P,

Regarding the non-integer frame rates, I proceeded by recommendations in the following link, relying on clock frequency.

This is working pretty well.

https://forums.xilinx.com/t5/Video-and-Audio/Does-Xilinx-HDMI-2-0-IP-support-non-integer-frame-rate-such-as/td-p/1073690

Boris Barak.

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
663 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

The reason why I mentioned custom resolution is I notice front porch/back porch timing becomes zero., is this expected or not?

0 Kudos
Highlighted
647 Views
Registered: ‎05-29-2019

Zero front porch/back porch timing is absolutely not expected to be.

We expect to get a right values according to CTA-861-G standard.

Driver waits to see the same values 2 sequential times, isn't it ?

See attached extraction from standard Table2:

1080p24: VIC=32, Hfront=638, Hsync=44, Hback=148, Vfront=4, Vsync=5, Vback=36;

I've just run the tests, see it took 622 frames to reach the right values (see in BLUE).

If you see in old test report (VTD_fluctuations.txt) the numbers are not zeros, but not make sense at all.

cnt=1136
WARN : HTotal: cur:prev 2686:2750
WARN : HActive: cur:prev 1920:1920
WARN : HSyncWidth: cur:prev 0044:0044
WARN : HFrontPorch: cur:prev 0639:0639
WARN : HBackPorch: cur:prev 0083:0147
WARN : F0PVTotal: cur:prev 0002:1276
WARN : F1VTotal: cur:prev 0001:0000
WARN : VActive: cur:prev 0000:1080
WARN : F0PVSyncWidth: cur:prev 0000:0000
WARN : F1VSyncWidth: cur:prev 0000:0000
WARN : F0PVFrontPorch: cur:prev 0002:0005
WARN : F1VFrontPorch: cur:prev 0001:0000
WARN : F0PVBackPorch: cur:prev 0000:0191
WARN : F1VBackPorch: cur:prev 0000:0000

Until that the driver don't enter into RxStreamUp state and don't call RxStreamUpCallback() function, the system stuck.

II've added an additional LOG functions, you've suggested in RxStreamUpCallback() function, see the result in RED.
cnt=620
WARN : HTotal: cur:prev 0000:0000
WARN : HActive: cur:prev 0000:0000
WARN : HSyncWidth: cur:prev 0000:0000
WARN : HFrontPorch: cur:prev 0000:0000
WARN : HBackPorch: cur:prev 0000:0000
WARN : F0PVTotal: cur:prev 0000:0000
WARN : F1VTotal: cur:prev 0000:0000
WARN : VActive: cur:prev 0000:0000
WARN : F0PVSyncWidth: cur:prev 0000:0000
WARN : F1VSyncWidth: cur:prev 0000:0000
WARN : F0PVFrontPorch: cur:prev 0000:0000
WARN : F1VFrontPorch: cur:prev 0000:0000
WARN : F0PVBackPorch: cur:prev 0000:0000
WARN : F1VBackPorch: cur:prev 0000:0000
cnt=621
WARN : HTotal: cur:prev 2750:0000
WARN : HActive: cur:prev 1920:0000
WARN : HSyncWidth: cur:prev 0044:0000
WARN : HFrontPorch: cur:prev 0638:0000
WARN : HBackPorch: cur:prev 0148:0000
WARN : F0PVTotal: cur:prev 1125:0000
WARN : F1VTotal: cur:prev 0000:0000
WARN : VActive: cur:prev 1080:0000
WARN : F0PVSyncWidth: cur:prev 0005:0000
WARN : F1VSyncWidth: cur:prev 0000:0000
WARN : F0PVFrontPorch: cur:prev 0004:0000
WARN : F1VFrontPorch: cur:prev 0000:0000
WARN : F0PVBackPorch: cur:prev 0036:0000
WARN : F1VBackPorch: cur:prev 0000:0000
cnt=622
WARN : HTotal: cur:prev 2750:2750
WARN : HActive: cur:prev 1920:1920
WARN : HSyncWidth: cur:prev 0044:0044
WARN : HFrontPorch: cur:prev 0638:0638
WARN : HBackPorch: cur:prev 0148:0148
WARN : F0PVTotal: cur:prev 1125:1125
WARN : F1VTotal: cur:prev 0000:0000
WARN : VActive: cur:prev 1080:1080
WARN : F0PVSyncWidth: cur:prev 0005:0005
WARN : F1VSyncWidth: cur:prev 0000:0000
WARN : F0PVFrontPorch: cur:prev 0004:0004
WARN : F1VFrontPorch: cur:prev 0000:0000
WARN : F0PVBackPorch: cur:prev 0036:0036
WARN : F1VBackPorch: cur:prev 0000:0000

====================================================================================
Color Format: RGB
Color Depth: 8
Pixels Per Clock: 2
Mode: Progressive
Frame Rate: 24Hz / 1.001
Resolution: 1920x1080@24Hz
INFO : Meas Pix clk: 74170000 Hz
RX: QPLL0
RX state: ready

QPLL0 settings
-------------
M : 1 - N : 160 - D : 16

RX MMCM settings
-------------
Mult : 20 - Div : 1 - Clk0Div : 40 - Clk1Div : 20 - Clk2Div : 40

DRU Settings
-------------
Version : 7
DRU is disabled

------------
HDMI RX SubSystem
------------

->HDMI RX Subsystem Cores
: HDMI RX
: HDCP 1.4 RX
: HDCP: AXIS Timer
HDMI RX version : 03.00 (0402)
HDCP 1.4 RX version : 00.01 (0209)

HDMI RX Mode - HDMI
------------
HDMI RX timing
------------
HSYNC Timing: hav=1920, hfp=638, hsw=44(hsp=1), hbp=148, htot=2750
VSYNC Timing: vav=1080, vfp=04, vsw=05(vsp=1), vbp=036, vtot=1125
VIC: 32
Scrambled: 0
Link quality
---------
Link quality channel 0 : excellent (0)
Link quality channel 1 : excellent (0)
Link quality channel 2 : excellent (0)
Audio
---------
Format : L-PCM
Channels : 8
ACR CTS : 74175
ACR N : 6144
Infoframe
---------
RX header: de000003

---------
HdmiRx State = UP
================================================================================

 

 

CTA-861-G_extract.png
0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
637 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

From your description, RX does not enter the stream-up state because the VTD data is invalid. It's waiting for the VTD to report sensible values.

Here is the Rx sequence, can you print out following information along with your frame counter, and check at which stage it takes very long time?

GT RX reset lock done (GTRXRESETDONE)

HdmiRxReadyCallback

start to check XV_HDMIRX_TMR_STA_OFFSET

enable audio and aux after link is stable

RX core reset is released

enable RX link

start to check XV_HDMIRX_TMR_STA_OFFSET

RX stream init, start to read video properties

start to check XV_HDMIRX_TMR_STA_OFFSET

RX Stream is armed

enable RX VTD

set RX status to lock

RX stream is up

 

0 Kudos
Highlighted
633 Views
Registered: ‎05-29-2019

Yes, of course.

We usually print this info during the tests, sorry for did not send you.

Here are all stages print, starting from re-connection, till the fluctuation on VTD happens:

INFO : Ch1: [VID-PHY] QPLL lost lock
INFO : Ch1: [HDMI-SS] RX Stream is Down
INFO : Ch1: [HDMI-SS] RX Stream is Down
INFO : [HdmiRx_PioIntrHandler:816] Ch1: Source is DVI
INFO : Ch1: [HDMI-SS] RX mode changed to DVI
INFO : Ch1: [VID-PHY] RX frequency event
INFO : Ch1: [VID-PHY] RX frequency event
INFO : Ch1: VphyHdmiRxInitCallback irq occured
INFO : Ch1: [HDMI-SS] RX TMDS reference clock change
INFO : Ch1: [VID-PHY] RX timer evenr
INFO : Ch1: [VID-PHY] RX DRU disable
INFO : Ch1: [VID-PHY] QPLL reconfig done
INFO : Ch1: [VID-PHY] GT RX reconfig start
INFO : Ch1: [VID-PHY] GT RX reconfig done
INFO : Ch1: [VID-PHY] QPLL lock
INFO : Ch1: [VID-PHY] RX reset done
INFO : Ch1: VphyHdmiRxReadyCallback irq occured
INFO : Ch1: [HDMI-SS] RX Stream Init
INFO : [HdmiRx_PioIntrHandler:704] Ch1: Link is ready
INFO : [HdmiRx_TmrIntrHandler:879] Ch1: HDMI counter event
INFO : [HdmiRx_TmrIntrHandler:887] Ch1: HDMI Stream idle state
INFO : Ch1: RxAudCallback irq occured
INFO : Audio channels: 8
INFO : Audio ARC CTS: 74250
INFO : Audio ARC N: 6144
INFO : [HdmiRx_TmrIntrHandler:879] Ch1: HDMI counter event
INFO : [HdmiRx_TmrIntrHandler:915] Ch1: HDMI Stream init state
INFO : [HdmiRx_PioIntrHandler:810] Ch1: Source is HDMI
INFO : Ch1: [HDMI-SS] RX mode changed to HDMI
INFO : Ch1: [VID-PHY] QPLL lost lock
INFO : Ch1: [VID-PHY] RX frequency event
INFO : Ch1: [VID-PHY] RX frequency event
INFO : Ch1: VphyHdmiRxInitCallback irq occured
INFO : Ch1: [HDMI-SS] RX TMDS reference clock change
INFO : [HdmiRx_TmrIntrHandler:879] Ch1: HDMI counter event
INFO : [HdmiRx_TmrIntrHandler:915] Ch1: HDMI Stream init state
INFO : Ch1: [HDMI-SS] RX Stream Start
INFO : Ch1: RX stream init
INFO : Ch1: [VID-PHY] RX MMCM reconfig done
INFO : Ch1: [VID-PHY] RX MMCM lock
INFO : Ch1: [VID-PHY] RX timer evenr
INFO : Ch1: [VID-PHY] RX DRU disable
INFO : Ch1: [VID-PHY] QPLL reconfig done
INFO : Ch1: [VID-PHY] GT RX reconfig start
INFO : Ch1: [VID-PHY] GT RX reconfig done
INFO : [HdmiRx_PioIntrHandler:720] Ch1: Video is ready
INFO : Ch1: [VID-PHY] QPLL lock
INFO : Ch1: [VID-PHY] RX reset done
INFO : Ch1: VphyHdmiRxReadyCallback irq occured
INFO : Ch1: [HDMI-SS] RX Stream Init
INFO : [HdmiRx_PioIntrHandler:704] Ch1: Link is ready
INFO : [HdmiRx_TmrIntrHandler:879] Ch1: HDMI counter event
INFO : [HdmiRx_TmrIntrHandler:887] Ch1: HDMI Stream idle state
INFO : [HdmiRx_TmrIntrHandler:879] Ch1: HDMI counter event
INFO : [HdmiRx_TmrIntrHandler:915] Ch1: HDMI Stream init state
INFO : Ch1: [HDMI-SS] RX Stream Start
INFO : Ch1: RX stream init
INFO : Ch1: [VID-PHY] RX MMCM reconfig done
INFO : Ch1: [VID-PHY] RX MMCM lock
INFO : [HdmiRx_PioIntrHandler:720] Ch1: Video is ready
INFO : [HdmiRx_TmrIntrHandler:879] Ch1: HDMI counter event
INFO : [HdmiRx_TmrIntrHandler:965] Ch1: HDMI Stream Arm state

 

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
625 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

From your debug information, it confirms the issue is related to VTD is trying to get the correct Rx Video Timing.

I will email you HDMI Rx IP with debug port, so we can check if the issue is in hardware as well, which Vivado version are you using?

0 Kudos
Highlighted
623 Views
Registered: ‎05-29-2019

Exactly, the issue is well focused to: VTD is trying to get the correct Rx Video Timing, as you've wrote.

It's happen often for some formats, while very rare for the others, strange.

For the time being we're using Vivado 2018.3 version.

Can you try help us as soon as possible.

We wouldn't want to see this rolled to the customers.

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
600 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

I just use EZMOVE to transfer two debug version IPs : one is used to debug the signals before driving the VTD. The other contains the debug signals around VTD.

 

0 Kudos
Highlighted
570 Views
Registered: ‎05-29-2019

Thank you very much.

Can you add a more detailed instructions on how to inject these debug IPs into current Vivado design, please. 

 

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
485 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

Step 1, unzip the debug version IP to a folder

Step 2: add the folder path to IP repository settings. You can refer to https://forums.xilinx.com/t5/Design-Entry/Adding-Creating-multiple-IPs-to-single-IP-Repository-in-Vivado/m-p/887906#M18131

Step 3: Highlight your hdmi IP in current project, right click -> Upgrade IP, and then reset generate IP output product. Then regenerate it, run synthesis。

Step 4: Open Synthesized design, run through setup debug flow, and then save synthesized design, run implementation.

The petalinux flow is the same.

0 Kudos
Highlighted
410 Views
Registered: ‎05-29-2019

Hi,

To let you know I had a short vacation and currently handling these test.

Will come back to you as soon as results will be available.

 

Best Regards.

Boris Barak.

0 Kudos
Highlighted
384 Views
Registered: ‎05-29-2019

Finally I could upgrade to hdmi_rx_ss core you sent me and generate 2 bitstreams.
First bitstream consist of v_hdmi_rx_v3_0_2018.3.
Second bitstream consist of v_hdmi_rx_v3_0_2018.3_vtd.
However the hardware target doesn't see any debug cores inside (see Vivado message below).
 
refresh_hw_device -update_hw_probes false [lindex [get_hw_devices xczu4_0] 0]
INFO: [Labtools 27-1434] Device xczu4 (JTAG device index = 0) is programmed with a design that has no supported debug core(s) in it.
 
Please your help.
Kind Regards.
Boris Barak.
0 Kudos
Highlighted
377 Views
Registered: ‎05-29-2019

Hi'

How I can verify that hdmi_rx_ss core is really upgraded to the on with debug port ?

The drivers reports me the same version in both cases:

->HDMI RX Subsystem Cores
: HDMI RX
: HDCP 1.4 RX
: HDCP: AXIS Timer
HDMI RX version : 03.00 (0405)
HDCP 1.4 RX version : 00.01 (0209)

Boris Barak.

0 Kudos
Highlighted
284 Views
Registered: ‎05-29-2019

Hi,

Finally I directly replaced IP in Vivado repository by the once you sent me.

Here:

\Xilinx\Vivado\2018.3\data\ip\xilinx

It seems that HDMI RX version has been changed, indeed.

The version for bot debug IP as follow:

However Hardware manager of Vivado still complain on debug core absence in FPGA.

INFO: [Labtools 27-1434] Device xczu4 (JTAG device index = 0) is programmed with a design that has no supported debug core(s) in it.

->HDMI RX Subsystem Cores
: HDMI RX
: HDCP 1.4 RX
: HDCP: AXIS Timer
HDMI RX version : 03.00 (0402)
HDCP 1.4 RX version : 00.01 (0209)

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
171 Views
Registered: ‎08-02-2007

Update : I have worked with Boris offline on this issue, and he was able to figure out the workaround in HDMI Rx driver. It's still waiting for QA

0 Kudos