06-17-2020 08:18 AM - edited 06-17-2020 02:41 PM
Hi, I am trying run my custom driver which allocate 512MiB CMA pool. I am using ZCU106 (zynq ultrascale+) FPGA and I am using 2019.2 linux. Whenever I run my application which doesnt even use ethernet but i still see kernel panic with "macb ff0e0000.ethernet eth0: DMA bus error: HRESP not OK"
[ 1405.481738] Unable to handle kernel paging request at virtual address ffffffe1456000c0
[ 1405.489651] Mem abort info:
[ 1405.492434] ESR = 0x96000145
[ 1405.495482] Exception class = DABT (current EL), IL = 32 bits
[ 1405.501393] SET = 0, FnV = 0
[ 1405.504431] EA = 0, S1PTW = 0
[ 1405.507555] Data abort info:
[ 1405.510421] ISV = 0, ISS = 0x00000145
[ 1405.514240] CM = 1, WnR = 1
[ 1405.517194] swapper pgtable: 4k pages, 39-bit VAs, pgdp = 0000000016306ff7
[ 1405.524059] [ffffffe1456000c0] pgd=0000000000000000, pud=0000000000000000
[ 1405.530840] Internal error: Oops: 96000145 [#1] SMP
[ 1405.535707] Modules linked in: dmaproxy(O) uio_pdrv_genirq
[ 1405.542229] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 4.19.0-xilinx-v2019.2 #1
[ 1405.550828] Hardware name: ZynqMP ZCU106 RevA (DT)
[ 1405.555605] pstate: 40000005 (nZcv daif -PAN -UAO)
[ 1405.560384] pc : __dma_inv_area+0x40/0x58
[ 1405.564381] lr : __swiotlb_unmap_page+0x74/0xa0
[ 1405.568893] sp : ffffff8008003d40
[ 1405.572192] x29: ffffff8008003d40 x28: ffffffc845ea07c0
[ 1405.577495] x27: ffffffc85ad4f100 x26: 0000000080c0c068
[ 1405.582799] x25: ffffff800caca180 x24: 2181b42100000000
[ 1405.588103] x23: 00000000456000c1 x22: ffffffc845ea47c0
[ 1405.593408] x21: 0000000000000600 x20: 2181b421456000c0
[ 1405.598712] x19: ffffffc85b020c10 x18: 0000000000000000
[ 1405.604015] x17: 0000000000000000 x16: 0000000000000000
[ 1405.609319] x15: 0000000000000000 x14: 0000000000000008
[ 1405.614623] x13: 0000000000000000 x12: 003d090000000000
[ 1405.619927] x11: 00003d08ffab34f8 x10: 0000000000000040
[ 1405.625232] x9 : ffffff800913ae08 x8 : ffffffc85b400268
[ 1405.630535] x7 : 0000000000000000 x6 : ffffffc85b400240
[ 1405.635840] x5 : 2181b421456000c0 x4 : 0000000000000000
[ 1405.641144] x3 : 000000000000003f x2 : 0000000000000040
[ 1405.646447] x1 : ffffffe1456006c0 x0 : ffffffe1456000c0
[ 1405.651753] Process swapper/0 (pid: 0, stack limit = 0x000000001dd560d9)
[ 1405.658436] Call trace:
[ 1405.660869] __dma_inv_area+0x40/0x58
[ 1405.664525] gem_rx+0x140/0x2d0
[ 1405.667657] macb_poll+0x54/0x108
[ 1405.670956] net_rx_action+0x104/0x2c0
[ 1405.674689] __do_softirq+0x104/0x1f8
[ 1405.678335] irq_exit+0x9c/0xb8
[ 1405.681470] __handle_domain_irq+0x64/0xb8
[ 1405.685556] gic_handle_irq+0x5c/0xb8
[ 1405.689202] el1_irq+0xb0/0x140
[ 1405.692329] cpuidle_enter_state+0x80/0x230
[ 1405.696503] cpuidle_enter+0x18/0x20
[ 1405.700062] call_cpuidle+0x1c/0x38
[ 1405.703534] do_idle+0x254/0x280
[ 1405.706746] cpu_startup_entry+0x20/0x28
[ 1405.710655] rest_init+0xac/0xb8
[ 1405.713875] start_kernel+0x3d0/0x3f8
[ 1405.717530] Code: 8a230000 54000060 d50b7e20 14000002 (d5087620)
[ 1405.723614] ---[ end trace 629b07839b178a45 ]---
[ 1405.728214] Kernel panic - not syncing: Fatal exception in interrupt
[ 1405.734552] SMP: stopping secondary CPUs
[ 1405.738458] Kernel Offset: disabled
[ 1405.741930] CPU features: 0x0,20802004
[ 1405.745661] Memory Limit: none
[ 1405.748702] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
[ 1406.497086] macb ff0e0000.ethernet eth0: DMA bus error: HRESP not OK
I am allocating CAM buffer for my application:
[ 0.000000] Reserved memory: created CMA memory pool at 0x0000000870000000, size 256 MiB
[ 0.000000] OF: reserved mem: initialized node devicebuffer@0x10000000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created CMA memory pool at 0x0000000860000000, size 256 MiB
[ 0.000000] OF: reserved mem: initialized node buffer@0, compatible id shared-dma-pool
Can anyone guide me what shall i do to for this error "macb ff0e0000.ethernet eth0: DMA bus error: HRESP not OK"?
I tried with changing CMA pool size, location but doesnt affect to this error. Not sure why I am seeing this error.
06-18-2020 09:10 AM
09-28-2020 04:15 AM
Any update about this issue?
We are having almost identical issue on 2019.1.
When transferring large data over ethernet, this kernel panic occurs. Same kernel panic when writing larger files (about 40 MB) to SD card while ethernet is on.
Workaround is to slow down ethernet transfers with sleeps and for SD card is to disable ethernet before writing, but this is really hacky.
09-28-2020 04:35 AM
@darko31Sorry, I didn't solve and didn't reach to bottom of this issue.
My particular case , I found, I was running out of CMA pool memory so by default Linux create 256MB pool and if you are allocating memory from this pool and trying to write large file it end up into crash. I would suggest check your memory layout for CMA pool. see how much memory used ? or as you said slow down ethernet traffic let SD card write enough time to write file so don't need much bigger size of pool space. ( As I said , I don't know exactly cause for problem but my case was slightly different as I was writing large debug file from my device driver..)
I am really keen to see if this can solve problem or if you find any solutions...….
09-29-2020 02:24 AM
It doesn't look like the CMA in my case, because I have three specific buffers reserved for udmabuf, and CMA by default is 256MB.
Mem info reports
CmaTotal: 262144 kB CmaFree: 261764 kB
Now for the SD write, for some inexplicable reason, 40MB is the limit which I can write before kernel panic occurs. I receive the file over ethernet, store it in a buffer, then write it to a file. If I write the file to the ramdisk first in /home partition, it works, up until the point where I call
cp /home/root/file /mnt/file
then the same kernel panic occurs. Only thing that works is if I disable ethernet by calling
ifconfig eth0 down
then copying successfully completes without kernel panic.
09-29-2020 03:57 AM
Actually, it probably was CMA. I've increased in kernel config from 256MB default value to 512MB and everything started working as expected.
No kernel panics, no HRESP not OK error.