cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
wgabor
Contributor
Contributor
1,996 Views
Registered: ‎01-22-2019

Zynq UltraScale+ SD card ADMA error, then USB fails too

Jump to solution

Dear Community,

We have a strange issue with a third party Zynq Ultracale+ board, with the 2018.x (where x=1 or 2) versions of the toolchain.

We compile the 2018.x version of the software tools (FSBL, PMU, BL31, device tree, u-boot) from Xilinx (without Yocto or PetaLinux), and Linux boots successfully. However on heavy SD card IO, such as with

dd if=/dev/mmcblk0p1 of=/dev/null bs=4M

there will be some errors in the kernel log:

[  181.069061] mmc0: ADMA error
[  181.071858] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[  181.078280] mmc0: sdhci: Sys addr:  0x00000000 | Version:  0x00001002
[  181.084702] mmc0: sdhci: Blk size:  0x00007200 | Blk cnt:  0x00000038
[  181.091125] mmc0: sdhci: Argument:  0x001f10d1 | Trn mode: 0x00000037
[  181.097549] mmc0: sdhci: Present:   0x01f70206 | Host ctl: 0x0000001f
[  181.103972] mmc0: sdhci: Power:     0x0000000f | Blk gap:  0x00000080
[  181.110395] mmc0: sdhci: Wake-up:   0x00000000 | Clock:    0x00000207
[  181.116818] mmc0: sdhci: Timeout:   0x00000004 | Int stat: 0x00000000
[  181.123241] mmc0: sdhci: Int enab:  0x02ff008b | Sig enab: 0x02ff008b
[  181.129665] mmc0: sdhci: AC12 err:  0x00000000 | Slot int: 0x00000000
[  181.136088] mmc0: sdhci: Caps:      0x31e8c881 | Caps_1:   0x00002007
[  181.142513] mmc0: sdhci: Cmd:       0x0000123a | Max curr: 0x00000000
[  181.148935] mmc0: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0x001e1d7f
[  181.155358] mmc0: sdhci: Resp[2]:   0x325b5900 | Resp[3]:  0x00000b00
[  181.161781] mmc0: sdhci: Host ctl2: 0x00000000
[  181.166208] mmc0: sdhci: ADMA Err:  0x00000001 | ADMA Ptr: 0x0000000070040218
[  181.173325] mmc0: sdhci: ============================================

Afterwards errors come on USB0 too:

[  181.179772] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint slot 1 ep 2
[  181.188174] xhci-hcd xhci-hcd.0.auto: @0000000070046de0 70066e60 00000000 0d003f88 01038000
[  181.196517] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint slot 1 ep 2
[  181.204925] xhci-hcd xhci-hcd.0.auto: @0000000070046e00 70066e70 00000000 0d003fa8 01038000
[  181.213257] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint slot 1 ep 2
[  181.221677] xhci-hcd xhci-hcd.0.auto: @0000000070046e10 70066e80 00000000 0d003f90 01038000
[  181.230009] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint slot 1 ep 2
[  181.238430] xhci-hcd xhci-hcd.0.auto: @0000000070046e20 70066e90 00000000 0d003f08 01038000
[  181.246762] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint slot 1 ep 2
[  181.255182] xhci-hcd xhci-hcd.0.auto: @0000000070046e30 70066ea0 00000000 0d003fa8 01038000
[  181.263514] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event for disabled endpoint slot 1 ep 2
[  181.271935] xhci-hcd xhci-hcd.0.auto: @0000000070046e40 70066eb0 00000000 0d003f88 01038000
[  181.280269] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 2 comp_code 13
[  181.290859] xhci-hcd xhci-hcd.0.auto: Looking for event-dma 0000000070066ed0 trb-start 0000000070066e60 trb-end 0000000070066e60 seg-start 0000000070066000 seg-end 0000000070066ff0
[  181.306919] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 2 comp_code 13
[  181.317506] xhci-hcd xhci-hcd.0.auto: Looking for event-dma 0000000070066ee0 trb-start 0000000070066e60 trb-end 0000000070066e60 seg-start 0000000070066000 seg-end 0000000070066ff0
[  181.333567] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 2 comp_code 13
[  181.344153] xhci-hcd xhci-hcd.0.auto: Looking for event-dma 0000000070066ef0 trb-start 0000000070066e60 trb-end 0000000070066e60 seg-start 0000000070066000 seg-end 0000000070066ff0

 

The two errors seem to be related, because they come in pair, I could not trigger them separately.

 

The related device tree parts:

 

/* SD0 */
               sdhci@ff160000 {
                       u-boot,dm-pre-reloc;
                       compatible = "xlnx,zynqmp-8.9a", "arasan,sdhci-8.9a";
                       status = "okay";
                       interrupt-parent = <0x4>;
                       interrupts = <0x0 0x30 0x4>;
                       reg = <0x0 0xff160000 0x0 0x1000>;
                       clock-names = "clk_xin", "clk_ahb";
                       xlnx,device_id = <0x0>;
                       #stream-id-cells = <0x1>;
                       iommus = <0x9 0x870>;
                       power-domains = <0x1c>;
                       clocks = <0x3 0x36 0x3 0x1f>;
                       disable-wp;
                       clock-frequency = <0xbebc200>;
                       xlnx,mio_bank = <0x0>;
               };

/* USB0 */
              usb0@ff9d0000 {
                       #address-cells = <0x2>;
                       #size-cells = <0x2>;
                       status = "okay";
                       compatible = "xlnx,zynqmp-dwc3";
                       reg = <0x0 0xff9d0000 0x0 0x100>;
                       clock-names = "bus_clk", "ref_clk";
                       power-domains = <0x26>;
                       ranges;
                       nvmem-cells = <0x19>;
                       nvmem-cell-names = "soc_revision";
                       clocks = <0x3 0x20 0x3 0x22>;
                       xlnx,usb-reset = <0x2faf080>;

                       dwc3@fe200000 {
                               compatible = "snps,dwc3";
                               status = "okay";
                               reg = <0x0 0xfe200000 0x0 0x40000>;
                               interrupt-parent = <0x4>;
                               interrupts = <0x0 0x41 0x4 0x0 0x45 0x4 0x0 0x4b 0x4>;
                               #stream-id-cells = <0x1>;
                               iommus = <0x9 0x860>;
                               snps,quirk-frame-length-adjustment = <0x20>;
                               snps,refclk_fladj;
                               snps,enable_guctl1_resume_quirk;
                               snps,enable_guctl1_ipd_quirk;
                               snps,xhci-stream-quirk;
                               dr_mode = "host";
                               snps,usb3_lpm_capable;
                               phy-names = "usb3-phy";
                               phys = <0x27 0x4 0x0 0x0 26000000>;
                               maximum-speed = "super-speed";
                       };
               };

I've tried adding "broken-mmc-highspeed" to the SD, also to set clock-frequency/max-frequency property, adding/removing the "no-1-8-v" property, but without success.

Also the SD card arrived with the board, it is brand new. I've also tested multiple SD card/multiple board combinations.

Am I missing something?

0 Kudos
1 Solution

Accepted Solutions
wgabor
Contributor
Contributor
1,383 Views
Registered: ‎01-22-2019

On a different board with different config I had the same issue, with 2019.2.

I have observed that when XSDK (Vitis) is running (and is connected to hw_server), this problem occurs. If not, it does not. 

Note: https://www.xilinx.com/support/answers/69143.html does not solve this issue.

View solution in original post

0 Kudos
4 Replies
wgabor
Contributor
Contributor
1,969 Views
Registered: ‎01-22-2019

Some addendum: with a Linux system built with PetaLinux, MMC and USB works correctly. Afterwards I've tried the following:

- fsbl, pmu, bl31, uboot comes from PetaLinux
- kernel and rootfs built manually

No error was triggered this way.

Other tries:

- fsbl, pmu, bl31 comes from PetaLinux

- kernel and rootfs, and uboot built manually 

The error is triggered this way.

So it seems that my u-boot configures the SoC incorrectly, and Linux does not update it correctly.

 

0 Kudos
wgabor
Contributor
Contributor
1,942 Views
Registered: ‎01-22-2019

Unfortunately still nothing, and I am out of ideas (checked PS configuration, FSBL, PMU, u-boot config, device tree, kernel config, SD card, but to no avail).

0 Kudos
wgabor
Contributor
Contributor
1,941 Views
Registered: ‎01-22-2019

And sometimes the kernel crashes in an interrupt too:

# dd if=/dev/mmcblk0 of=/dev/null bs=1M
[ 130.426480] mmc0: ADMA error
[ 130.429416] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[ 130.436035] mmc0: sdhci: Sys addr: 0x00000000 | Version: 0x00001002
[ 130.442658] mmc0: sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000380
[ 130.449283] mmc0: sdhci: Argument: 0x0002bc00 | Trn mode: 0x00000037
[ 130.455904] mmc0: sdhci: Present: 0x01f70206 | Host ctl: 0x0000001b
[ 130.462526] mmc0: sdhci: Power: 0x0000000f | Blk gap: 0x00000080
[ 130.469148] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x00000607
[ 130.475774] mmc0: sdhci: Timeout: 0x00000004 | Int stat: 0x00000000
[ 130.482400] mmc0: sdhci: Int enab: 0x02ff008b | Sig enab: 0x02ff008b
[ 130.489020] mmc0: sdhci: AC12 err: 0x00000000 | Slot int: 0x00000000
[ 130.495643] mmc0: sdhci: Caps: 0x31ecc881 | Caps_1: 0x00002007
[ 130.502262] mmc0: sdhci: Cmd: 0x0000123a | Max curr: 0x00000000
[ 130.508891] mmc0: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0x003a437f
[ 130.515511] mmc0: sdhci: Resp[2]: 0x325b5900 | Resp[3]: 0x00000b00
[ 130.522130] mmc0: sdhci: Host ctl2: 0x00000000
[ 130.526666] mmc0: sdhci: ADMA Err: 0x00000001 | ADMA Ptr: 0x000000007004020c
[ 130.534011] mmc0: sdhci: ============================================
[ 130.540662] Unable to handle kernel paging request at virtual address ffffff80090ee004
[ 130.548827] Mem abort info:
[ 130.551644] Exception class = DABT (current EL), IL = 32 bits
[ 130.557726] SET = 0, FnV = 0
[ 130.565295] EA = 0, S1PTW = 0
[ 130.572925] Data abort info:
[ 130.580275] ISV = 0, ISS = 0x00000007
[ 130.588454] CM = 0, WnR = 0
[ 130.595607] swapper pgtable: 4k pages, 39-bit VAs, pgd = ffffff8009052000
[ 130.606747] [ffffff80090ee004] *pgd=000000087fffe003, *pud=000000087fffe003, *pmd=000000087fffa003, *pte=0000000000000000
[ 130.626441] Internal error: Oops: 96000007 [#1] SMP
[ 130.635615] Modules linked in:
[ 130.653228] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 4.14.0-xilinx-v2018.1 #2
[ 130.671146] Hardware name:
[ 130.681742] task: ffffff8008da2100 task.stack: ffffff8008d90000
[ 130.692717] PC is at sdhci_irq+0x480/0x908
[ 130.701873] LR is at sdhci_irq+0x46c/0x908
[ 130.710983] pc : [<ffffff8008769f98>] lr : [<ffffff8008769f84>] pstate: 800001c5
[ 130.728869] sp : ffffff8008003df0
[ 130.737429] x29: ffffff8008003df0 x28: 0000000000000000
[ 130.748092] x27: 0000000002000000 x26: ffffffc87a917600
[ 130.758807] x25: 0000000002ff003f x24: 0000000000000000
[ 130.769437] x23: 00000000fd006e00 x22: 00000000000f0001
[ 130.780002] x21: 000000000270003e x20: ffffffc87b01c600
[ 130.790468] x19: 0000000000000010 x18: 0000000000000010
[ 130.800772] x17: 0000000000000000 x16: 0000000000000000
[ 130.810858] x15: 0000000000000001 x14: ffffff8008aa6a58
[ 130.820896] x13: 0000000000000000 x12: 00000000ffffffff
[ 130.830869] x11: 0000000000000010 x10: 0000000000000020
[ 130.840728] x9 : ffffffc877dd69b0 x8 : 0000000000000001
[ 130.850605] x7 : ffffff800e103620 x6 : 000000000000015e
[ 130.860469] x5 : 0000000000000001 x4 : 0000000002008000
[ 130.870277] x3 : ffffff80090ee010 x2 : 0000000000000000
[ 130.880087] x1 : fffffffffffffff4 x0 : 000000000000000c
[ 130.889729] Process swapper/0 (pid: 0, stack limit = 0xffffff8008d90000)
[ 130.900813] Call trace:
[ 130.907443] Exception stack(0xffffff8008003cb0 to 0xffffff8008003df0)
[ 130.918247] 3ca0: 000000000000000c fffffffffffffff4
[ 130.934563] 3cc0: 0000000000000000 ffffff80090ee010 0000000002008000 0000000000000001
[ 130.950947] 3ce0: 000000000000015e ffffff800e103620 0000000000000001 ffffffc877dd69b0
[ 130.967597] 3d00: 0000000000000020 0000000000000010 00000000ffffffff 0000000000000000
[ 130.984719] 3d20: ffffff8008aa6a58 0000000000000001 0000000000000000 0000000000000000
[ 131.002121] 3d40: 0000000000000010 0000000000000010 ffffffc87b01c600 000000000270003e
[ 131.020005] 3d60: 00000000000f0001 00000000fd006e00 0000000000000000 0000000002ff003f
[ 131.038554] 3d80: ffffffc87a917600 0000000002000000 0000000000000000 ffffff8008003df0
[ 131.057882] 3da0: ffffff8008769f84 ffffff8008003df0 ffffff8008769f98 00000000800001c5
[ 131.077951] 3dc0: 000000000000015d ffffff800e0f9620 0000008000000000 ffffff8008769f84
[ 131.098868] 3de0: ffffff8008003df0 ffffff8008769f98
[ 131.110517] [<ffffff8008769f98>] sdhci_irq+0x480/0x908
[ 131.122479] [<ffffff80080db884>] __handle_irq_event_percpu+0x5c/0x148
[ 131.135893] [<ffffff80080db98c>] handle_irq_event_percpu+0x1c/0x58
[ 131.149128] [<ffffff80080dba0c>] handle_irq_event+0x44/0x78
[ 131.161748] [<ffffff80080df6c8>] handle_fasteoi_irq+0xa0/0x178
[ 131.174658] [<ffffff80080da9c4>] generic_handle_irq+0x24/0x38
[ 131.187338] [<ffffff80080db05c>] __handle_domain_irq+0x5c/0xb0
[ 131.200055] [<ffffff80080814e4>] gic_handle_irq+0x64/0xc0
[ 131.212249] Exception stack(0xffffff8008d93d80 to 0xffffff8008d93ec0)
[ 131.225526] 3d80: 0000000000000000 ffffff8008d96b80 00000000004d75aa 0000000000000000
[ 131.247030] 3da0: 000000000ccccccd 00ffffffffffffff 00000001a0bf4e9e 0000000000000001
[ 131.268553] 3dc0: 00000000000001c0 ffffff8008082000 0000000000001000 0000000000000000
[ 131.290071] 3de0: 0000000034d5d91d 0000004877202000 ffffff8008da2100 0000000000000000
[ 131.311609] 3e00: 0000000000000000 0000000000000000 0000000000000000 0000001e5e06049c
[ 131.333177] 3e20: 0000000000000001 ffffffc87a94e000 0000000000000001 ffffffc87ba13400
[ 131.354716] 3e40: 0000001e5de0a968 ffffffc87ba13400 ffffff8008da2100 0000000000000400
[ 131.376237] 3e60: 0000000000d10018 ffffff8008d93ec0 ffffff800874fdd0 ffffff8008d93ec0
[ 131.397776] 3e80: ffffff800874fe10 0000000040000145 ffffffc87ba13460 0000000000000001
[ 131.419318] 3ea0: ffffffffffffffff 0000000000000001 ffffff8008d93ec0 ffffff800874fe10
[ 131.440958] [<ffffff80080830f0>] el1_irq+0xb0/0x140
[ 131.452788] [<ffffff800874fe10>] cpuidle_enter_state+0x158/0x1f8
[ 131.465806] [<ffffff800874fee8>] cpuidle_enter+0x18/0x20
[ 131.478041] [<ffffff80080cfb0c>] call_cpuidle+0x1c/0x38
[ 131.490172] [<ffffff80080cfd50>] do_idle+0x1a0/0x1d8
[ 131.502042] [<ffffff80080cfef8>] cpu_startup_entry+0x20/0x28
[ 131.514712] [<ffffff8008a07b54>] rest_init+0xac/0xb8
[ 131.526648] [<ffffff8008d10b78>] start_kernel+0x398/0x3ac
[ 131.538989] Code: b94087a4 f94047a3 cb0003e1 8b000063 (78616862)
[ 131.552036] ---[ end trace 377c93c4939dca8c ]---
[ 131.563548] Kernel panic - not syncing: Fatal exception in interrupt
[ 131.576799] SMP: stopping secondary CPUs
[ 131.587477] Kernel Offset: disabled
[ 131.597525] CPU features: 0x002004
[ 131.607282] Memory Limit: none
[ 131.616522] ---[ end Kernel panic - not syncing: Fatal exception in interrupt

0 Kudos
wgabor
Contributor
Contributor
1,384 Views
Registered: ‎01-22-2019

On a different board with different config I had the same issue, with 2019.2.

I have observed that when XSDK (Vitis) is running (and is connected to hw_server), this problem occurs. If not, it does not. 

Note: https://www.xilinx.com/support/answers/69143.html does not solve this issue.

View solution in original post

0 Kudos