cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted
1,878 Views
Registered: ‎05-29-2019

VID_Phy/Hdmi_rx_ss drivers run reliability issue

Hello,

We are using Zynq xczu4ev-fbvb900-1-e chip in our HDMI RX system solution.

There are 2 RX HDMI channels in our Zynq appliance. 

The Xilinx code (https://github.com/Xilinx/embeddedsw/tree/master/XilinxProcessorIPLib/drivers) for VID_Phy/Hdmi_rx_ss is running in Linux user space on APU-PS.

We are experienceing an issues during realibility test.

The test: HDMI source switches to different video formats periodically (10 sec).

The software compares the source format with the one discovered after RX stream is up (like in the example below the format matchs).

INFO : [RxStreamUpCallback1:1328] RX stream is up
INFO : Mode: HDMI
Color Format: YUV_422
Color Depth: 12
Pixels Per Clock: 2
Mode: Progressive
Frame Rate: 60Hz
Resolution: 1920x1080@60Hz


In the case it doesn't match we log an error.

In most of cases when input format was changed - HDMI driver wasn't capable to recognize the change and internal driver state machine was stopped in the middle, waiting for something.

When signal will be changed next time with high probability drivers will detect it correctly.

We tried a different modes of operation, but error rate is still high.

Even running only single RX channel (closing all other sw processes) we got 2 errors after 500 iterations of the test.

When running the whole user application, including 2 RX HDMI the error rate close to 10% !!!

I'm wondering if running RX HDMI drivers on some "bare metal" processor could help on this issue.

Please urgent help.  

Best Regards.

Boris Barak.

Tags (2)
0 Kudos
54 Replies
Highlighted
Xilinx Employee
Xilinx Employee
1,663 Views
Registered: ‎08-02-2007

Hi boris.barak@vitec.com 

Can you check if you see this problem in bera-metal design, please?

Can you provide terminal log when HDMI driver wasn't capable to recognize the change and internal driver state machine was stopped in the middle?

0 Kudos
Highlighted
1,651 Views
Registered: ‎05-29-2019

Hi,

We couldn't check this code with bare-metal option.

Actually PL part of Zynq Ultrascale is very tough in aspect of LUTs.

Therefore any idea to use MicroBlaze couldn't be applied.

Attached log of the drivers up to stuck point.

We are considering use the drivers to the Linux Kernel space.

Could it dramatically improve performance (however Linux is not Real Time). ?

Could you point us to the mature Linux drivers, please.

 

Best Regards.

Boris Barak.

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
1,642 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

Please refer to HDMI Rx linux driver page : https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841884/Xilinx+V4L2+hdmirx+driver -> Kernel Configuration Options for Driver to figure out how to set u configure HDMI driver.

FYI : HDMI drivers can be get from : https://github.com/Xilinx/hdmi-modules

 

 

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
1,607 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

The other thing I want to double check is which branch do you use. Which vivado version are you targeting at? I need to ensure your Vivado version matches with driver version.

As the problem is stuck at Video PHY RX MMCM config, can you print out all the Video register values? It can give me more hints on where might be the problem

0 Kudos
Highlighted
1,601 Views
Registered: ‎05-29-2019

Vivado version used is 2018.3.1

Actually system could stuck at different points.

It can be either PHY or MAC step.

Stuck in "Video PHY RX MMCM config" is the specific case, that was logged.

Is it makes sense go back to the test and sample PHY/MAC registers every time the system stuck ?

We are using 2 hdmi PHY/MAC cores in the same FPGA, for 2 hdmi Rx channels.

BTW: The error rate(stuck)  is increasing while running both channels drivers in Linux user space (vs one channel only).

It emphesises a real time issue as the root of the problen, isn't it ?

There are few relatively big "sleep" states in the callback functions

I'm wondering if real time issue could be solved, while finally running drivers of both hdmi cores in Linux Kernel space.

Best regards.

Boris Barak.

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
1,564 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

Can you share the recipe file used to add HDMI kernel module? I want to double check if you use the correct driver version.

It's very important to dump all the Video PHY register file. 

When issue occurs, please read out following information :

cat /sys/devices/platform/amba_pl\@0/<device_addr>.v_hdmi_rx_ss/hdmi_info

cat /sys/devices/platform/amba_pl\@0/<device_addr>.v_hdmi_rx_ss/vphy_log

cat /sys/devices/platform/amba_pl\@0/<device_addr>.v_hdmi_rx_ss/vphy_info

 

0 Kudos
Highlighted
1,532 Views
Registered: ‎05-29-2019

Hi

Stuck at Video PHY RX MMCM config has been reproduced right now.

Attached PHY and HDMI_RX_SS registers dump at that point .

Attached also recipe file prepared, to be used in HDMI kernel module and Vivado snapshot (we did not applied Kernel module yet).

Could you review this, please ?

I'm quite confused with clock name aliases here.

Unfortunately I did not find log pathes, you've menthioned in your response.

Should it appear after Kernel modules run only ?

Best Regards.

Boris Barak.

 

Vivdo_Address_Editor.png
0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
1,499 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

From XVPHY_CLKDET_FREQ_RX_REG 0x00000210, 0x046CE800 = 74.25Mhz. It doesn't seem to be correct to me.  

I have done a quick test at my end. The pixel clock for 1080p60 should be 148.5MHz.

HDMI_Rx.JPG

Can you double check this? I will be on vacation for 3 weeks. During this time, @aoifem will follow up this.

0 Kudos
Highlighted
1,466 Views
Registered: ‎05-29-2019

Hi,

Attached PHY registers dump for the case of normal lock and stuck.

The test was after changing input source as follow : 1920x1080p60 => 1920x1080p59.94.

Attached also LOG of all interrupts happened since input reconnect till stream is Up (normal flow).

Here VPHY/HDMI-RX-SS interrupts happen alternately.

Is it normal ?

We can see the huge number of Aux interrupt events happen.

In the current case we get 117 Aux interrupts of 135 Total (about 87% !!!!!!).

I'm wondering if HDMI-RX-SS interrupt should be processed at all, before VPHY operation is finished.

Best Regards.

Boris Barak.

 

 

0 Kudos
Highlighted
Moderator
Moderator
1,307 Views
Registered: ‎11-21-2018

Thansk boris.barak@vitec.com 

 

I'm looking into this and I'll get back to you as soon as I can. 

 

Regards, 

Aoife
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
Highlighted
1,281 Views
Registered: ‎05-29-2019

Hi,

The error rate had been reduced dramatically after we applied interrupts enable/disable.

See attached file with the functions borrowed.

However there are still errors exists (the error rate is about 0.27% in long run).

In such case we get "Video timings do not match!!!".

See attached log of the system (we change the source format automatically every 5 seconds).

Best Regards.

Boris Barak.

0 Kudos
Highlighted
Moderator
Moderator
1,244 Views
Registered: ‎11-21-2018

Hi boris.barak@vitec.com 

Thank you for the update. I think this case is more complicated than anticipated, so I will need more time to look into this.

Please note that our offices will be closed until the 2nd of Jan., so either myself or @xud will try to respond to you after that. 

Hope you have a nice winter break! 

Regards, 

Aoife
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
Highlighted
1,158 Views
Registered: ‎05-29-2019

Hi,

I wish you a happy new year.

Some additional information collected.

Sometimes VTD couldn't relax for a long number of iterations.

See attached file with the number of interations (cnt) and the VTD parameters.

Thus XV_HdmiRx_GetVideoTiming() function doesn't return OK and the State Machine did not go to the STREAM_UP state.

 

Best Regards.

Boris Barak.

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
1,126 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

Happy New Year!

From the interrupt log, I can see you get "[VID-PHY] Ch1: RX reset done INFO : [VphyHdmiRxReadyCallback1:1564] VphyHdmiRxReadyCallback irq occured"

It means Video PHY is set properly. I can see your HDMI Rx is ARMed

What happens between HDMI Rx Arm and HDMI Rx Stream up is

enable RX VTD

set RX status to lock

 

It seems that you can't get VTD lock. I will double check your Video Timing in VTD_fluctuation file, and then get back to you.

Highlighted
Xilinx Employee
Xilinx Employee
1,081 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

Boris, the Video Timing in VTD_fluctuations is not correct.

According to CEA-861-F protocol, Htotal for 1920p60 should be 2200, but it's 2750. There are also other errors in your Video PHY.

That's why Video Bridge can't be locked. What HDMI Source do you use to generate Video stream?

 

0 Kudos
Highlighted
1,070 Views
Registered: ‎05-29-2019

Hi,

Actually the source format is 1080p24 in this case, so 2750 seems OK.

The problem that VTD couldn't stabilize from time to time (in many other cases this source working well).

We are using our own video generator device in the tests (it was approved in other projects).

https://www.vitec.com/product/Extensor-VPG-70

Anyway, do you think that physical layer of this specific HDMI sync device under test lacks stability ?

Boris Barak.

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
1,065 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

Yes, I can see sometimes the video timing is all zeros.

When you report HDMI Rx info, what's the link quality? Do you have ZCU106 board, and see if this board has the same issue? There is onboard TMDS181 can improve the link quality.

If you want to debug further, I  can send you an IP that can check if there is HDMI Rx sync problem.

0 Kudos
Highlighted
1,058 Views
Registered: ‎05-29-2019

Hi,

Sometimes it takes a lot of iterations for link quality to be excellent (see below).

Unfortunately we don't have ZCU106 board here.

We have PI3HDX1204EZHEX re-driver on our product board.

Do you recommend trying to optimize re-driver parameters.

Do you have an IP to be sure that link is perfectly optimized ?

I'm currently asked to finish HDCP1.4 subcore debug in order to make a demo of the product.

Then I'll be back working on stability issues.

Link quality channel 0 : bad (65535)
Link quality channel 1 : bad (65535)
Link quality channel 2 : bad (65535)

Link quality channel 0 : average (15686)
Link quality channel 1 : average (15946)
Link quality channel 2 : average (16194)

Link quality channel 0 : average (2354)
Link quality channel 1 : average (2358)
Link quality channel 2 : average (2358)

............................................................

............................................................

............................................................

Link quality channel 0 : excellent (0)
Link quality channel 1 : excellent (0)
Link quality channel 2 : excellent (0)

 

Boris Barak.

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
1,048 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

Only excellent means there is no link quality problem, Depending on how many errors you have, you may see good, average or bad, all of them means there is a problem with link.

0 Kudos
Highlighted
904 Views
Registered: ‎05-29-2019

Hi,

I turned back to the system detection realibility test.

After feeding an HDMI source with different set-up of formats, running in loop (every 15 seconds source changes a format), some conclusion can be made.

The common errors are the fluctuations of the interface, with difficulties to relax of VTD (VideoTimingDetector).

It takes about 10-11 iterations to relax in normal case.

As can be seen from the first attached file – it took 606 iterations and a long time of about 31.5 sec to relax.

As can be seen from the second attached file – it couldn’t be relaxed and after 598 iterations QPLL lost lock.

[VID-PHY] Ch1: RX MMCM lock

[VID-PHY] Ch1: QPLL lost lock

After QPLL reconfiguration and all other states processing of PHY and HDMI-RX, the relaxation has been achived fast.

See below the summary of the last tests performed.

It's difficult to discover a common denumenator of the "bad" formats, that distuingush it from the "better" ones.

The souce is Vitec VPG70 generator and reputed as reliable one on other projects.

******************************************************************************************************************************************

  1. For the following set-up of formats, running in loop with 15 seconds gap between it, the detection error rate is about 0.02% (~6 errors on 3000 iterations).

VPG_vs_7k_formats = [

    '1280x720p50',

    '1920x1080i50',

    '1920x1080p50',

    '1920x1080i59_9',

    '1280x720p59_9',

    '1920x1080p59_9',

    '1280x720p60',

    '1920x1080p60',

    '1920x1080i60']

 

  1. For the following set-up of formats, running in loop with 15 seconds gap between it, the detection error rate is about 0.04% (~4 errors on 1000 iterations).

VPG_vs_7k_formats = [

    '1280x720p50',

    '1920x1080i50',

    '1920x1080p50',

    '1920x1080i59_9',

    '1280x720p59_9',

    '1920x1080p59_9',

    '1280x720p60',

    '1920x1080p60',

    '1920x1080i60',

    '720x576i50',

    '720x576p50',

    '720x487i60',

    '720x487p60']

 

  1. For the following set-up of formats, running in loop with 15 seconds gap between it, the detection error rate is about 4% ((~4 errors on 100 iterations).

VPG_vs_7k_formats = [

    '1920x1080p25',

    '1920x1080p30',

    '1920x1080p29_9',

    '1920x1080p23_9',

    '1920x1080p24' ]

 

 

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
889 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

If you loopback the HDMI Rx and HDMI Tx of Vitec VPG70 generator, do you see this issue?

I used to see an issue if I use script to automatic switch between different color formats on my QD780D, there was some errors.

Manually switch doesn't have this problem.Firstly we need to roll out the issue caused by HDMI Source itself.

0 Kudos
Highlighted
879 Views
Registered: ‎05-29-2019

Actually VPG70 has no HDMI inputs to make a loopback.

I've currently tried 50 manual HDMI cable reconnections from VPG70 and PC source as well.

No any issues in detection have been happened.

Obviously manual cable reconnections leads to another state machine flow, compared with the automatic tests (see below in bold).

Have you tried a script to automatic switch between different color formats of QD780D with a "problematic" formats, menthioned in my previous post ?

Are you observing a long fluctuations of  "Timing Detector" without relaxation ?

************************************************************************************

[HDMI-RX] Ch1: RX Stream is Up
INFO : [RxStreamUpCallback1:1786] RX stream is up
Color Format: YUV_422
Color Depth: 12
Pixels Per Clock: 2
Mode: Progressive
Frame Rate: 24Hz
Resolution: 1920x1080@24Hz
[VID-PHY] Ch1: QPLL lost lock
[VID-PHY] Ch1: RX frequency event
INFO : [HdmiRx_PioIntrHandler:688] Ch1: cable is disconnected
[HDMI-RX] Ch1: RX cable is disconnected
[HDMI-RX] Ch1: RX Stream is Down
INFO : [RxStreamDownCallback1:2206] RxStreamDownCallback irq occured
[HDMI-RX] Ch1: RX Stream is Down
INFO : [RxStreamDownCallback1:2206] RxStreamDownCallback irq occured
INFO : [HdmiRx_PioIntrHandler:816] Ch1: Source is DVI
[HDMI-RX] Ch1: RX mode changed to DVI
INFO : [HdmiRx_PioIntrHandler:681] Ch1: Cable is connected
[HDMI-RX] Ch1: RX cable is connected
[VID-PHY] Ch1: RX frequency event
INFO : [VphyHdmiRxInitCallback1:2149] VphyHdmiRxInitCallback irq occured
[HDMI-RX] Ch1: RX TMDS reference clock change
[VID-PHY] Ch1: RX timer event
[VID-PHY] Ch1: RX DRU disable
[VID-PHY] Ch1: QPLL reconfig done
[VID-PHY] Ch1: GT RX reconfig start
[VID-PHY] Ch1: GT RX reconfig done
[VID-PHY] Ch1: QPLL lock
[VID-PHY] Ch1: RX reset done
INFO : [VphyHdmiRxReadyCallback1:2173] VphyHdmiRxReadyCallback irq occured
[HDMI-RX] Ch1: RX Stream Init
INFO : [HdmiRx_PioIntrHandler:704] Ch1: Link is ready
INFO : [HdmiRx_PioIntrHandler:704] Ch1: Link is ready
INFO : [HdmiRx_TmrIntrHandler:879] Ch1: HDMI counter event
INFO : [HdmiRx_TmrIntrHandler:887] Ch1: HDMI Stream idle state
[XV_HdmiRx_IntrHandler:229] Ch1: Interrupt occured: HDMI-SS Audio
INFO : [RxAudCallback1:2238] RxAudCallback irq occured
INFO : Audio channels: 8
INFO : Audio ARC CTS: 74250
INFO : Audio ARC N: 6144
INFO : [HdmiRx_PioIntrHandler:810] Ch1: Source is HDMI
[HDMI-RX] Ch1: RX mode changed to HDMI
INFO : [HdmiRx_TmrIntrHandler:879] Ch1: HDMI counter event
INFO : [HdmiRx_TmrIntrHandler:915] Ch1: HDMI Stream init state
[HDMI-RX] Ch1: RX Stream Start
INFO : [RxStreamInitCallback1:2078] RX stream init
[VID-PHY] Ch1: RX MMCM reconfig done
[VID-PHY] Ch1: RX MMCM lock
INFO : [HdmiRx_PioIntrHandler:720] Ch1: Video is ready
INFO : [HdmiRx_TmrIntrHandler:879] Ch1: HDMI counter event
INFO : [HdmiRx_TmrIntrHandler:965] Ch1: HDMI Stream Arm state
cnt=0
cnt=1
cnt=2
cnt=3
cnt=4
cnt=5
cnt=6
cnt=7
cnt=8
cnt=9
cnt=10
cnt=11
cnt=12
cnt=13
cnt=14
[HDMI-RX] Ch1: RX Stream is Up
INFO : [RxStreamUpCallback1:1786] RX stream is up
Color Format: YUV_422
Color Depth: 12
Pixels Per Clock: 2
Mode: Progressive
Frame Rate: 25Hz
Resolution: 1920x1080@25Hz

***********************************************************************************

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
872 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

If manually switching is fine, it means the problem is to do with HDMI Generator.

For that QD780 problem, we used to work with QuantumData, and try to get patch on latest firmware to resolve the problem.

Firstly you can try to upgrade to the latest version of Vitec Firmware, if it doesn't work, I'm afraid you need to contact Vitec support team.

0 Kudos
Highlighted
867 Views
Registered: ‎05-29-2019

Thank you.

Did you mean Quantum, but not Vitec in the following sentense ?

"Firstly you can try to upgrade to the latest version of Vitec Firmware, if it doesn't work, I'm afraid you need to contact Vitec support team."

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
861 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

As your HDMI source is from Vitec, I mean Vitec.

When I mention manual switch, it doesn't mean you need to unplug/pulg the cable, I mean you can change the color format(or frame rate) on HDMI Generator panel.

The other thing you can try is to adjust Frequency lock counter threshold value in Clock Detector (HDMI) Control Register (0x0200) of Video PHY, and see if makes any difference.

 
0 Kudos
Highlighted
850 Views
Registered: ‎05-29-2019

It's OK, actually our company called Vitec and VPG70 is our product.

I double checked with our support team about latest firmware  and also replaced the unit to be sure.

Actually changing the color format(or frame rate) on HDMI Generator panel have the same behaviour like an automatic control.

We would try also tests with QD780D for reference.

I'll try to calibrate Clock Detector (HDMI) Control Register (0x0200) and will come backto you with the answers.

0 Kudos
Highlighted
801 Views
Registered: ‎05-29-2019

Hi 

I'm trying to play with "FreqLockThreshold" PHY parameter, by doing changes in XVphy_Hdmi_CfgInitialize() function.

See below the XVPHY_CLKDET_CTRL_REG readback, after different treshold settings.

Is it makes sense ?

Can you suggest possible treshold setting that could resolve our problem, please ?

***********************************************************************

//Default

XVphy_ClkDetSetFreqLockThreshold(InstancePtr, 40); 
XVPHY_CLKDET_CTRL_REG @0x00000200 - 0x00000501

***********************************************************************

XVphy_ClkDetSetFreqLockThreshold(InstancePtr, 80);
XVPHY_CLKDET_CTRL_REG @0x00000200 - 0x00000F01

***********************************************************************

XVphy_ClkDetSetFreqLockThreshold(InstancePtr, 20);
XVPHY_CLKDET_CTRL_REG @0x00000200 - 0x00000781

***********************************************************************

 

0 Kudos
Highlighted
697 Views
Registered: ‎05-29-2019

Hi,

 

It seems changing FreqLockThreshold did not change the results.

Please suggest the possible values of this parameters to improve it.

Are any other ideas possible ?

Boris Barak.

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
691 Views
Registered: ‎08-02-2007

boris.barak@vitec.com 

The FreqThreshold register only helps if Rx freq isn't always lock.

I notice you are trying the change frame rate from 60 to 60/1.001. In this case, I'm not sure if RX can be reset properly after switching. I need to do some test on this at my end, and then get back to you.