UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Observer ptg
Observer
816 Views
Registered: ‎05-07-2018

Synchronous External Abort on Rx ISR of AXI QSPI

Jump to solution

I'm suffering an abort when my ZynqMP board communicates with MCU.

The board is based on XCZU4EG and the SPI IP is AXI Quad SPI (normal mode, AXI Lite).

This occurs after running around 1 hour, though it completely works as I expected until then.
I can't find any other similar case like this and really hard to figure it out because it's occurring in IRQ handler.
It always occurs on xspi_read_rx_fifo_8.

If there's an solution it would be great but for the beginning I'd like to know how to debug aborts like this.

 

[ 5027.549336] Synchronous External Abort: synchronous external abort (0x96000210) at 0xffffff800b9d006c                                                      
[ 5027.558486] Internal error: : 96000210 [#1] SMP                                                                                                                          
[ 5027.562995] Modules linked in: mali(O)                                                                                                                                   
[ 5027.566730] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O    4.14.0 #1                                                                           
[ 5027.575318] Hardware name: ZynqMP (DT)                                                                                                                             
[ 5027.579572] task: ffffff8008df1a00 task.stack: ffffff8008de0000                                                                                                          
[ 5027.585481] PC is at xspi_read_rx_fifo_8+0x28/0x60                                                                                                                       
[ 5027.590249] LR is at xilinx_spi_irq+0x90/0xa8                                                                                                                            
[ 5027.594589] pc : [<ffffff80085f3220>] lr : [<ffffff80085f3878>] pstate: 200001c5                                                                                         
[ 5027.601966] sp : ffffff8008003e50                                                                                                                                        
[ 5027.605265] x29: ffffff8008003e50 x28: ffffff8008df1a00                                                                                                                  
[ 5027.610559] x27: 0000000000000001 x26: ffffff8008c2bc48                                                                                                                  
[ 5027.615854] x25: ffffff8008e72588 x24: ffffffc87a975a00                                                                                                                  
[ 5027.621148] x23: 0000000000000034 x22: 0000000000000004                                                                                                                  
[ 5027.626443] x21: 0000000000000001 x20: ffffffc87a2b6800                                                                                                                  
[ 5027.631738] x19: ffffffc87a2b6dc8 x18: 0000000000040e00                                                                                                                  
[ 5027.637033] x17: 00000000009530b8 x16: ffffff80081c0bb0                                                                                                                  
[ 5027.642328] x15: 000000000000087d x14: 0000000000000020                                                                                                                  
[ 5027.647623] x13: 0000000000000000 x12: 0000000000000040                                                                                                                  
[ 5027.652917] x11: 0000000000000018 x10: 0000000000000034                                                                                                                  
[ 5027.658213] x9 : ffffffc87ac00408 x8 : 0000000000000000                                                                                                                  
[ 5027.663508] x7 : ffffffc87a975a00 x6 : ffffffc87a975a00                                                                                                                  
[ 5027.668802] x5 : ffffffc87ac00240 x4 : 0000000000000010                                                                                                                  
[ 5027.674097] x3 : ffffffc86fd4d020 x2 : 0000000000000003                                                                                                                  
[ 5027.679392] x1 : ffffff800b9d006c x0 : ffffffc87a2b6dc8                                                                                                                  
[ 5027.684688] Process swapper/0 (pid: 0, stack limit = 0xffffff8008de0000)                                                                                                 
[ 5027.691371] Call trace:                                                                                                                                                  
[ 5027.693803] Exception stack(0xffffff8008003d10 to 0xffffff8008003e50)                                                                                                    
[ 5027.700226] 3d00:                                   ffffffc87a2b6dc8 ffffff800b9d006c                                                                                    
[ 5027.708039] 3d20: 0000000000000003 ffffffc86fd4d020 0000000000000010 ffffffc87ac00240                                                                                    
[ 5027.715850] 3d40: ffffffc87a975a00 ffffffc87a975a00 0000000000000000 ffffffc87ac00408                                                                                    
[ 5027.723662] 3d60: 0000000000000034 0000000000000018 0000000000000040 0000000000000000                                                                                    
[ 5027.731474] 3d80: 0000000000000020 000000000000087d ffffff80081c0bb0 00000000009530b8                                                                                    
[ 5027.739287] 3da0: 0000000000040e00 ffffffc87a2b6dc8 ffffffc87a2b6800 0000000000000001                                                                                    
[ 5027.747098] 3dc0: 0000000000000004 0000000000000034 ffffffc87a975a00 ffffff8008e72588                                                                                    
[ 5027.751039] spidev spi2.0: SPI transfer timed out                                                                                                                        
[ 5027.755044] spidev spi0.0: SPI transfer timed out                                                                                                                        
[ 5027.756225] spidev spi1.0: SPI transfer timed out                                                                                                                        
[ 5027.768976] 3de0: ffffff8008c2bc48 0000000000000001 ffffff8008df1a00 ffffff8008003e50                                                                                    
[ 5027.776785] 3e00: ffffff80085f3878 ffffff8008003e50 ffffff80085f3220 00000000200001c5                                                                                    
[ 5027.784597] 3e20: ffffffc87a5e4d20 ffffffc87a8f3600 0000008000000000 ffffff80080817f4                                                                                    
[ 5027.792408] 3e40: ffffff8008003e50 ffffff80085f3220                                                                                                                      
[ 5027.797273] [<ffffff80085f3220>] xspi_read_rx_fifo_8+0x28/0x60                                                                                                           
[ 5027.803088] [<ffffff80080df984>] __handle_irq_event_percpu+0x9c/0x128                                                                                                    
[ 5027.809509] [<ffffff80080dfa2c>] handle_irq_event_percpu+0x1c/0x58                                                                                                       
[ 5027.815672] [<ffffff80080dfaac>] handle_irq_event+0x44/0x78                                                                                                              
[ 5027.821227] [<ffffff80080e3724>] handle_fasteoi_irq+0x9c/0x190                                                                                                           
[ 5027.827043] [<ffffff80080dea84>] generic_handle_irq+0x24/0x38                                                                                                            
[ 5027.832771] [<ffffff80080df104>] __handle_domain_irq+0x5c/0xb8                                                                                                           
[ 5027.838587] [<ffffff80080814c0>] gic_handle_irq+0x68/0xc8                                                                                                                
[ 5027.843967] Exception stack(0xffffff8008de3d80 to 0xffffff8008de3ec0)                                                                                                    
[ 5027.850392] 3d80: 0000000000000000 ffffffc87ff6ff00 000000487719c000 000004000386ac9f                                                                                    
[ 5027.858204] 3da0: 0000000000000016 00ffffffffffffff 0000002236408007 00000000000000ae                                                                                    
[ 5027.866016] 3dc0: 000000000000005f ffffffc87ff6ef04 ffffffc87ff6eee4 0000000000000018                                                                                    
[ 5027.873828] 3de0: 071c71c71c71c71c 0000000000000000 0000000000000020 000000000000087d                                                                                    
[ 5027.881640] 3e00: ffffff80081c0bb0 00000000009530b8 0000000000040e00 00000492914a59eb                                                                                    
[ 5027.889452] 3e20: 0000000000000000 ffffffc87aba1c00 0000000000000000 ffffffc879848400                                                                                    
[ 5027.897264] 3e40: 00000492914a1c29 ffffff8008df1a00 000000007fee77e0 0000000000000400                                                                                    
[ 5027.905076] 3e60: 0000000000d60018 ffffff8008de3ec0 ffffff800875ae80 ffffff8008de3ec0                                                                                    
[ 5027.912888] 3e80: ffffff800875ae84 0000000060000145 ffffffc879848400 0000000000000000                                                                                    
[ 5027.920700] 3ea0: ffffffffffffffff 0000000000000000 ffffff8008de3ec0 ffffff800875ae84                                                                                    
[ 5027.928512] [<ffffff80080830f0>] el1_irq+0xb0/0x140                                                                                                                      
[ 5027.933375] [<ffffff800875ae84>] cpuidle_enter_state+0x154/0x200                                                                                                         
[ 5027.939363] [<ffffff800875af68>] cpuidle_enter+0x18/0x20                                                                                                                 
[ 5027.944658] [<ffffff80080d3470>] call_cpuidle+0x18/0x30                                                                                                                  
[ 5027.949866] [<ffffff80080d36ac>] do_idle+0x19c/0x1d8                                                                                                                     
[ 5027.954813] [<ffffff80080d3850>] cpu_startup_entry+0x20/0x28                                                                                                             
[ 5027.960457] [<ffffff8008a1eccc>] rest_init+0xac/0xb8                                                                                                                     
[ 5027.965404] [<ffffff8008d60b70>] start_kernel+0x384/0x398                                                                                                                
[ 5027.970786] Code: 5400018d d2800002 f9400001 9101b021 (b9400021)                                                                                                         
[ 5027.976861] ---[ end trace 9ed4b8846e516106 ]---                                                                                                                         
[ 5027.981461] Kernel panic - not syncing: Fatal exception in interrupt                                                                                                     
[ 5027.987798] SMP: stopping secondary CPUs                                                                                                                                 
[ 5027.991703] Kernel Offset: disabled                                                                                                                                      
[ 5027.995175] CPU features: 0x002004                                                                                                                                       
[ 5027.998559] Memory Limit: none                                                                                                                                           
[ 5028.001599] Rebooting in 1 seconds..

 

0 Kudos
1 Solution

Accepted Solutions
Highlighted
Observer ptg
Observer
631 Views
Registered: ‎05-07-2018

Re: Synchronous External Abort on Rx ISR of AXI QSPI

Jump to solution

@gudishak Recently I've managed to get its stability. Here's an explanation of background for a reference.

An userspace daemon runs on Linux and communicates with MCU 333 times per second. The unit of message is around ~500 bytes, split in every 64 bytes, and full-duplex.
I found that the QSPI's FIFO Depth config was set to 16 (default value) which seems to be too short for my usage. The "transaction width" config is widely-used 8 bit. Thus, The FIFO size should be 16 bytes. A part of message (64 bytes) can easily fill it and IRQ should raise too frequent.
So then I increased the FIFO Depth to 256 and the result was positive. No aborts happened even after running 40+ hours.

I have no JTAG cables so I can't analyze this bug at the moment but I'm still curious about it.

5 Replies
Moderator
Moderator
696 Views
Registered: ‎06-27-2017

Re: Synchronous External Abort on Rx ISR of AXI QSPI

Jump to solution

Hi @ptg,

Could you please share the device tree changes for AXI QSPI?

Best Regards
Kranthi
--------------------------
Don't forget to reply, kudo, and accept as solution.
0 Kudos
Observer ptg
Observer
680 Views
Registered: ‎05-07-2018

Re: Synchronous External Abort on Rx ISR of AXI QSPI

Jump to solution

 @gudishak Here is the corresponding device tree node.

There are more 3 nodes mostly identical to it except 'reg'.

        axi_quad_spi@b0006000 {
            bits-per-word = <0x8>;
            compatible = "xlnx,xps-spi-2.00.a";
            fifo-size = <0x10>;
            interrupt-names = "ip2intc_irpt";
            interrupt-parent = <0x4>;
            interrupts = <0x0 0x6c 0x1>;
            num-cs = <0x1>;
            reg = <0x0 0xb0006000 0x0 0x1000>;
            xlnx,num-ss-bits = <0x1>;
            xlnx,spi-mode = <0x0>;
            #address-cells = <0x1>;
            #size-cells = <0x0>;

            spidev@0 {
                compatible = "spidev";
                spi-max-frequency = <0x1e8480>;
                reg = <0x0>;
            };
        };
0 Kudos
Moderator
Moderator
652 Views
Registered: ‎06-27-2017

Re: Synchronous External Abort on Rx ISR of AXI QSPI

Jump to solution

Hi @ptg,

Could you please try to read the status register from XSCT? if not possible try to read before the crash in the interrupt handler to check whether Under run or not.

 

Best Regards
Kranthi
--------------------------
Don't forget to reply, kudo, and accept as solution.
0 Kudos
Highlighted
Observer ptg
Observer
632 Views
Registered: ‎05-07-2018

Re: Synchronous External Abort on Rx ISR of AXI QSPI

Jump to solution

@gudishak Recently I've managed to get its stability. Here's an explanation of background for a reference.

An userspace daemon runs on Linux and communicates with MCU 333 times per second. The unit of message is around ~500 bytes, split in every 64 bytes, and full-duplex.
I found that the QSPI's FIFO Depth config was set to 16 (default value) which seems to be too short for my usage. The "transaction width" config is widely-used 8 bit. Thus, The FIFO size should be 16 bytes. A part of message (64 bytes) can easily fill it and IRQ should raise too frequent.
So then I increased the FIFO Depth to 256 and the result was positive. No aborts happened even after running 40+ hours.

I have no JTAG cables so I can't analyze this bug at the moment but I'm still curious about it.

Moderator
Moderator
622 Views
Registered: ‎06-27-2017

Re: Synchronous External Abort on Rx ISR of AXI QSPI

Jump to solution

@ptg,

Thanks for the detailed explanation. I will write a Answer record for the same, it will be very useful for the folks who looking for same issue.

 

 

Best Regards
Kranthi
--------------------------
Don't forget to reply, kudo, and accept as solution.
0 Kudos