UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Contributor
Contributor
472 Views
Registered: ‎05-14-2018

Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

Hello,

I am using a xczu7ev-fbvb900-1-e with Vivado 2019.1 with a block design consisting of a UHD-SDI GT block with a single Rx and Tx channel, and the accompanying frame buffer read/write blocks. When either the SDI kernel module or I (with devmem) write to the upper half of the address range for the Tx subsystem block (which the kernel module does in order to reset the internal VTC block), the kernel panics with the following message:

# devmem 0xb0030000 32 0

[ 2228.651842] SError Interrupt on CPU3, code 0xbf000002 -- SError
[ 2228.651845] CPU: 3 PID: 2139 Comm: devmem Tainted: G           O      4.19.0-xilinx-v2019.1 #1
[ 2228.651846] Hardware name: xlnx,zynqmp (DT)
[ 2228.651848] pstate: 60000000 (nZCv daif -PAN -UAO)
[ 2228.651849] pc : 0000007fa82b028c
[ 2228.651850] lr : 0000007fa82b03f4
[ 2228.651851] sp : 0000007fed4b4ac0
[ 2228.651852] x29: 0000007fed4b4ac0 x28: 0000000000000001 
[ 2228.651856] x27: 0000007fa83e5000 x26: 0000007fa83e2b28 
[ 2228.651859] x25: 0000000000000001 x24: 0000007fa83e15c8 
[ 2228.651862] x23: 0000007fa84f9710 x22: 0000007fa83e0000 
[ 2228.651865] x21: 0000000000000000 x20: 0000007fa83e0000 
[ 2228.651867] x19: 0000007fa83e5ce8 x18: 0000000000000100 
[ 2228.651870] x17: 0000007fa82b0418 x16: 0000005572fac4f0 
[ 2228.651873] x15: 0000007fa827cac8 x14: 00000000000003f3 
[ 2228.651876] x13: 0000000000000000 x12: 0000000000000000 
[ 2228.651878] x11: ffffffffffffffff x10: 0000000000000008 
[ 2228.651881] x9 : 0000000000000007 x8 : 00000000000000de 
[ 2228.651884] x7 : 0000007fa8385c60 x6 : 0000000000000000 
[ 2228.651886] x5 : 00000000b0030000 x4 : 0000000000000000 
[ 2228.651889] x3 : 0000000000000000 x2 : 0000000000000001 
[ 2228.651892] x1 : 0000000000000000 x0 : 0000000000000001 
[ 2228.651895] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 2228.651897] CPU: 3 PID: 2139 Comm: devmem Tainted: G           O      4.19.0-xilinx-v2019.1 #1
[ 2228.651899] Hardware name: xlnx,zynqmp (DT)
[ 2228.651900] Call trace:
[ 2228.651901]  dump_backtrace+0x0/0x148
[ 2228.651902]  show_stack+0x14/0x20
[ 2228.651903]  dump_stack+0x90/0xb4
[ 2228.651904]  panic+0x120/0x268
[ 2228.651905]  nmi_panic+0x6c/0x70
[ 2228.651906]  arm64_serror_panic+0x74/0x80
[ 2228.651908]  is_valid_bugaddr+0x0/0x8
[ 2228.651909]  el0_error_naked+0x10/0x18
[ 2228.651918] SMP: stopping secondary CPUs
[ 2228.651920] Kernel Offset: disabled
[ 2228.651921] CPU features: 0x0,20802004
[ 2228.651922] Memory Limit: none
[ 2228.828315] ---[ end Kernel panic - not syncing: Asynchronous SError Interrupt ]---

Reading from this location shows a "Bus Error" warning with devmem.

The address map in Vivado is as follows:

Screenshot_20191022_162215.png

 

I've tried attaching a system ILA to the AXI bus, which shows a SLVERR when the read or write is carried out (after quite a few clock cycles - maybe a timeout?). All writes and reads to the lower 64k of the address range work perfectly fine, so the whole block isn't in reset; it seems to be a problem with the internal VTC IP.

Screenshot_20191022_163256.png

I have also tried recreating the project, using different AXI ports on the Zynq, and using different addresses, with no change. The VTC IP has been created and is visibly connected to the AXI crossbar in the implemented design schematic.

 

Any tips on what to try at this point would be very appreciated.

 

Cheers,

Isaac

0 Kudos
1 Solution

Accepted Solutions
Contributor
Contributor
81 Views
Registered: ‎05-14-2018

Re: Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

Hi @florentw ,

I believe I have found the problem. It seems to have been something to do with the timing of the resets coming into the cmp_gt_ctrl[63:0] vector of the block. I originally had custom RTL module which fed the appropriate resets and ready signals into the UHD-SDI block, but it seems that enabling everything at once, or not toggling a reset signal, caused some kind of issue with the txoutclk. I have replaced the module with the compact_gt_ctrl.v file from the 2019.1 TRD and attached the fmc_init_done signal to the s_axi_aresetn signal, and now the txoutclk is active and I can access the registers of the internal VTC.

I have confirmed that the outputs of both my custom module and the compact_gt_ctrl module are the same (and the QPLL's are locked with both modules), so it must be an ordering or timing issue.

Thank you for your help in narrowing down the issue.

Cheers,

Isaac

0 Kudos
14 Replies
Moderator
Moderator
395 Views
Registered: ‎11-09-2015

Re: Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

HI @isaacjt 

How are you connecting the ZynqMP to the UHD-SDI? Is it through an AXI interconnect?

If yes, could you try to replace it with a Smartconnect IP?

I have seen this type of issues with the AXI Interconnect IP.

Regards,


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
Contributor
Contributor
389 Views
Registered: ‎05-14-2018

Re: Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

Hi @florentw ,

Yes I am using an AXI Interconnect with the ZynqMP, but I have also tried a Smartconnect and even just a plain AXI protocol converter (AXI4 <-> AXI4LITE), all with the same result.

Cheers,

Isaac

0 Kudos
Moderator
Moderator
387 Views
Registered: ‎11-09-2015

Re: Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

Hi @isaacjt 

Can you try to set the address range to 64K for the UHD-SDI TX Subsystem? Make sure you hit validate block design each time you do a change


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
Contributor
Contributor
381 Views
Registered: ‎05-14-2018

Re: Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

Hi @florentw ,

 

when I try that, I get the following error:

set_property range 64K [get_bd_addr_segs {zynq_ultra_ps_e_0/Data/SEG_v_smpte_uhdsdi_tx_ss_0_Reg}]
ERROR: [BD 41-70] Exec TCL: </sdi_subsys/v_smpte_uhdsdi_tx_ss_0/S_AXI_CTRL/Reg> is a hierarchical segment with a range of <128K>. Its range cannot be modified to <64K> because lower level scoped segments could be hidden..

I also cannot make the address range bigger:

set_property range 256K [get_bd_addr_segs {zynq_ultra_ps_e_0/Data/SEG_v_smpte_uhdsdi_tx_ss_0_Reg}]
ERROR: [BD 41-70] Exec TCL: proposed address <0xB004_0000 [ 256K ]> does not conform to the range limitations <0x0000_0000 [ 128K ]> for this fixed segment </sdi_subsys/v_smpte_uhdsdi_tx_ss_0/S_AXI_CTRL/Reg>.

Thank you for your advice so far,

Isaac

0 Kudos
Moderator
Moderator
372 Views
Registered: ‎11-09-2015

Re: Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

Hi @isaacjt 

Do you have a SDI source connected and transmitting data when you are getting the kernel panic? The VTC can generate a Slave error in 2 conditions: if the reset is aserted or if the video clock is not present but clken is high.

Thus it might help to have a video source connected


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
Contributor
Contributor
368 Views
Registered: ‎05-14-2018

Re: Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

Hi @florentw ,

no, I was hoping to loopback the signal sent out on Tx (using the gstreamer kmssink example from https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/115998765/Zynq+UltraScale+MPSoC+VCU+TRD+2019.1+-+SDI+Video+Display#ZynqUltraScale%EF%BC%8BMPSoCVCUTRD2019.1-SDIVideoDisplay-4AppendixB ) back into the Rx in order to test if the Rx can receive the signal and correctly interpret it.

Is it not possible to transmit a signal without an incoming signal? This could be a problem for us as we had intended to have 2x Rx channels and 4x Tx channels...

I will give it another go with another video source connected to the Rx.

Cheers,

Isaac

0 Kudos
Moderator
Moderator
362 Views
Registered: ‎11-09-2015

Re: Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

HI @isaacjt 

This is a debugging process. I am not sure if the UHD-SDI TX under linux can or cannot work without a source connected. But this could give us a hint on the root cause of the Slave error.

It could be that the clock is propagated from RX to TX as it is a path-through. You would need to redesign your system to have TX and RX fully independant

 

 


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
Contributor
Contributor
272 Views
Registered: ‎05-14-2018

Re: Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

Hi @florentw ,

On second thought I don't think this can be the case, as I've successfully generated an SDI signal on the ZCU106 dev board from a file *without* an incoming SDI signal using the 2019.1 SDI TRD. This block design has a similar setup to the TRD design.

Unfortunately I won't be able to debug this further for another week, but after that I will try to attach more ILA probes to various parts of the system and see what I can find out.

Cheers,

Isaac

Scholar watari
Scholar
255 Views
Registered: ‎06-16-2013

Re: Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

Hi @isaacjt 

 

I'm not familiar with UHD-SDI Tx and aarch64 eabi.

But, according to your kernel panic message, it seems be occred to access "Error Register" via user program.

(Refer the value of x5 and x10 and make sure the meaning of "el0".(*1))

 

Also, I suggest you to investigate aarch64 eabi.

This definition is pointed out the meaning of each registers and there are any hint to debug software.

 

*1)

elX (Exception Level)

el0 : User program

el1 : OS

el2 : hyper byzer

el3 : secure monitor

 

Best regards,

 

0 Kudos
Moderator
Moderator
228 Views
Registered: ‎11-09-2015

Re: Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

@isaacjt wrote:

Hi @florentw ,

On second thought I don't think this can be the case, as I've successfully generated an SDI signal on the ZCU106 dev board from a file *without* an incoming SDI signal using the 2019.1 SDI TRD. This block design has a similar setup to the TRD design.

Unfortunately I won't be able to debug this further for another week, but after that I will try to attach more ILA probes to various parts of the system and see what I can find out.

Cheers,

Isaac


If adding a probe, I would add on the video clock going to the VTC. (the easier is to add some logic to check the frequency to make sure it is active). You can also probe directly at the VTC output (on the syntesized design to add ILA on the VTC directly, refer to Video Series 31 – Debugging a Video System using an ILA )


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
Contributor
Contributor
156 Views
Registered: ‎05-14-2018

Re: Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

Hi @florentw,

It seems that the txoutclk (which goes to the VTC) is not working. I've attached an ILA to the clock as you suggested, and I see the following:

Screenshot_20191107_153431.png

Net1 is the rxoutclk from the UHD-SDI GT block, which seems to be working fine. The cmp_gt_sts bits 0 & 1 show that the QPLLs are locked.

The lack of a clock going into the VTC would definitely explain why it doesn't respond on the AXI interface.

The UHD-SDI GT block is configured as follows:

Screenshot_20191107_153848.png

I will keep digging deeper to try and find out why the clock is not active.

Cheers,

Isaac

0 Kudos
Moderator
Moderator
148 Views
Registered: ‎11-09-2015

Re: Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

Hi @isaacjt 

Great. We seem to be on the correct path.

From the signal cmp_gt_sts[63:0], the QPLL0 seems to be locked so I would expect to have the txoutclk running

Regards


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos
Contributor
Contributor
82 Views
Registered: ‎05-14-2018

Re: Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

Hi @florentw ,

I believe I have found the problem. It seems to have been something to do with the timing of the resets coming into the cmp_gt_ctrl[63:0] vector of the block. I originally had custom RTL module which fed the appropriate resets and ready signals into the UHD-SDI block, but it seems that enabling everything at once, or not toggling a reset signal, caused some kind of issue with the txoutclk. I have replaced the module with the compact_gt_ctrl.v file from the 2019.1 TRD and attached the fmc_init_done signal to the s_axi_aresetn signal, and now the txoutclk is active and I can access the registers of the internal VTC.

I have confirmed that the outputs of both my custom module and the compact_gt_ctrl module are the same (and the QPLL's are locked with both modules), so it must be an ordering or timing issue.

Thank you for your help in narrowing down the issue.

Cheers,

Isaac

0 Kudos
Moderator
Moderator
74 Views
Registered: ‎11-09-2015

Re: Kernel panic when writing to upper 64k (VTC) of the SMTPE UHD-SDI Tx Subsystem

Jump to solution

HI @isaacjt 

Great. Well done in finding the issue

Regards


Florent
Product Application Engineer - Xilinx Technical Support EMEA
**~ Don't forget to reply, give kudos, and accept as solution.~**
0 Kudos