cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
systemsdeveloper
Explorer
Explorer
827 Views
Registered: ‎06-10-2019

eth0: DMA bus error: HRESP not OK with 2019.2

Hi, I am trying run my custom driver which allocate 512MiB CMA pool. I am using ZCU106 (zynq ultrascale+) FPGA and I am using 2019.2 linux. Whenever I  run my application which doesnt even use ethernet but i still see kernel panic with "macb ff0e0000.ethernet eth0: DMA bus error: HRESP not OK"

Error message:

[ 1405.481738] Unable to handle kernel paging request at virtual address ffffffe1456000c0
[ 1405.489651] Mem abort info:
[ 1405.492434] ESR = 0x96000145
[ 1405.495482] Exception class = DABT (current EL), IL = 32 bits
[ 1405.501393] SET = 0, FnV = 0
[ 1405.504431] EA = 0, S1PTW = 0
[ 1405.507555] Data abort info:
[ 1405.510421] ISV = 0, ISS = 0x00000145
[ 1405.514240] CM = 1, WnR = 1
[ 1405.517194] swapper pgtable: 4k pages, 39-bit VAs, pgdp = 0000000016306ff7
[ 1405.524059] [ffffffe1456000c0] pgd=0000000000000000, pud=0000000000000000
[ 1405.530840] Internal error: Oops: 96000145 [#1] SMP
[ 1405.535707] Modules linked in: dmaproxy(O) uio_pdrv_genirq
[ 1405.542229] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 4.19.0-xilinx-v2019.2 #1
[ 1405.550828] Hardware name: ZynqMP ZCU106 RevA (DT)
[ 1405.555605] pstate: 40000005 (nZcv daif -PAN -UAO)
[ 1405.560384] pc : __dma_inv_area+0x40/0x58
[ 1405.564381] lr : __swiotlb_unmap_page+0x74/0xa0
[ 1405.568893] sp : ffffff8008003d40
[ 1405.572192] x29: ffffff8008003d40 x28: ffffffc845ea07c0
[ 1405.577495] x27: ffffffc85ad4f100 x26: 0000000080c0c068
[ 1405.582799] x25: ffffff800caca180 x24: 2181b42100000000
[ 1405.588103] x23: 00000000456000c1 x22: ffffffc845ea47c0
[ 1405.593408] x21: 0000000000000600 x20: 2181b421456000c0
[ 1405.598712] x19: ffffffc85b020c10 x18: 0000000000000000
[ 1405.604015] x17: 0000000000000000 x16: 0000000000000000
[ 1405.609319] x15: 0000000000000000 x14: 0000000000000008
[ 1405.614623] x13: 0000000000000000 x12: 003d090000000000
[ 1405.619927] x11: 00003d08ffab34f8 x10: 0000000000000040
[ 1405.625232] x9 : ffffff800913ae08 x8 : ffffffc85b400268
[ 1405.630535] x7 : 0000000000000000 x6 : ffffffc85b400240
[ 1405.635840] x5 : 2181b421456000c0 x4 : 0000000000000000
[ 1405.641144] x3 : 000000000000003f x2 : 0000000000000040
[ 1405.646447] x1 : ffffffe1456006c0 x0 : ffffffe1456000c0
[ 1405.651753] Process swapper/0 (pid: 0, stack limit = 0x000000001dd560d9)
[ 1405.658436] Call trace:
[ 1405.660869] __dma_inv_area+0x40/0x58
[ 1405.664525] gem_rx+0x140/0x2d0
[ 1405.667657] macb_poll+0x54/0x108
[ 1405.670956] net_rx_action+0x104/0x2c0
[ 1405.674689] __do_softirq+0x104/0x1f8
[ 1405.678335] irq_exit+0x9c/0xb8
[ 1405.681470] __handle_domain_irq+0x64/0xb8
[ 1405.685556] gic_handle_irq+0x5c/0xb8
[ 1405.689202] el1_irq+0xb0/0x140
[ 1405.692329] cpuidle_enter_state+0x80/0x230
[ 1405.696503] cpuidle_enter+0x18/0x20
[ 1405.700062] call_cpuidle+0x1c/0x38
[ 1405.703534] do_idle+0x254/0x280
[ 1405.706746] cpu_startup_entry+0x20/0x28
[ 1405.710655] rest_init+0xac/0xb8
[ 1405.713875] start_kernel+0x3d0/0x3f8
[ 1405.717530] Code: 8a230000 54000060 d50b7e20 14000002 (d5087620)
[ 1405.723614] ---[ end trace 629b07839b178a45 ]---
[ 1405.728214] Kernel panic - not syncing: Fatal exception in interrupt
[ 1405.734552] SMP: stopping secondary CPUs
[ 1405.738458] Kernel Offset: disabled
[ 1405.741930] CPU features: 0x0,20802004
[ 1405.745661] Memory Limit: none
[ 1405.748702] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
[ 1406.497086] macb ff0e0000.ethernet eth0: DMA bus error: HRESP not OK

 

I am allocating CAM buffer for my application:

[ 0.000000] Reserved memory: created CMA memory pool at 0x0000000870000000, size 256 MiB
[ 0.000000] OF: reserved mem: initialized node devicebuffer@0x10000000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created CMA memory pool at 0x0000000860000000, size 256 MiB
[ 0.000000] OF: reserved mem: initialized node buffer@0, compatible id shared-dma-pool

 

Can anyone guide me what shall i do to for this error "macb ff0e0000.ethernet eth0: DMA bus error: HRESP not OK"?

I tried with changing CMA pool size, location but doesnt affect to this error. Not sure why I am seeing this error.

Thanks,

0 Kudos
Reply
6 Replies
systemsdeveloper
Explorer
Explorer
761 Views
Registered: ‎06-10-2019

I get same error if try to write big files on SDcard. 

0 Kudos
Reply
systemsdeveloper
Explorer
Explorer
757 Views
Registered: ‎06-10-2019

I found same kind of error https://forums.xilinx.com/t5/Embedded-Linux/Ethernet-DMA-Bus-Error/td-p/1072970

I cant see any proper solution @shabbirk any suggestion?

0 Kudos
Reply
darko31
Observer
Observer
560 Views
Registered: ‎05-07-2018

Any update about this issue?

We are having almost identical issue on 2019.1.

When transferring large data over ethernet, this kernel panic occurs. Same kernel panic when writing larger files (about 40 MB) to SD card while ethernet is on.

Workaround is to slow down ethernet transfers with sleeps and for SD card is to disable ethernet before writing, but this is really hacky.

0 Kudos
Reply
systemsdeveloper
Explorer
Explorer
547 Views
Registered: ‎06-10-2019

@darko31Sorry, I didn't solve and didn't reach to bottom of this issue. 

My particular case , I found,  I was running out of CMA pool memory so by default Linux create 256MB pool and if you are allocating memory from this pool and trying to write large file it end up into crash. I would suggest check your memory layout for CMA pool. see how much memory used ? or as you said slow down ethernet traffic let SD card write enough time to write file so don't need much bigger size of pool space. ( As I said , I don't know exactly cause for problem but my case was slightly different as I was writing large debug file from my device driver..)

I am really keen to see if this can solve problem or if you find any solutions...….

darko31
Observer
Observer
518 Views
Registered: ‎05-07-2018

It doesn't look like the CMA in my case, because I have three specific buffers reserved for udmabuf, and CMA by default is 256MB.

Mem info reports

CmaTotal:         262144 kB
CmaFree:          261764 kB

 

Now for the SD write, for some inexplicable reason, 40MB is the limit which I can write before kernel panic occurs. I receive the file over ethernet, store it in a buffer, then write it to a file. If I write the file to the ramdisk first in /home partition, it works, up until the point where I call

cp /home/root/file /mnt/file 

 

then the same kernel panic occurs. Only thing that works is if I disable ethernet by calling

ifconfig eth0 down

then copying successfully completes without kernel panic.

0 Kudos
Reply
darko31
Observer
Observer
504 Views
Registered: ‎05-07-2018

Actually, it probably was CMA. I've increased in kernel config from 256MB default value to 512MB and everything started working as expected.

No kernel panics, no HRESP not OK error.

0 Kudos
Reply