UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Observer jcteiwes
Observer
10,587 Views
Registered: ‎01-23-2011

linux-xlnx kernel oopses when doing heavy IO with SDHC on Zynq XC7Z030

Hi forum,

 

We are using a custom board which is running fine except when doing IO on the SDHC controller. This can be when installing new packages via apt or copying large files to the SD-Card. 

The system is running linaro on top of kernel linux-xlnx-3.9. The bug can be repoduced quite good with running 

 

dd if=/dev/zero of=pySPACEcenter/storage/test2.eeg bs=1MB count=1000

 

which stalls the system after ~130MB of data copied.

 

Here is the section of the syslog:

 


 

[ 76.287560] ------------[ cut here ]------------
[ 76.292094] kernel BUG at include/linux/mm.h:445!
[ 76.296794] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
[ 76.302600] Modules linked in: ipv6 gpio_xilinx_axi_custom xadc
[ 76.308499] CPU: 0 Not tainted (3.9.0-xilinx-dirty #20)
[ 76.314065] PC is at __rmqueue+0x2e8/0x338
[ 76.318139] LR is at __rmqueue+0xc4/0x338
[ 76.322138] pc : [<c007ec18>] lr : [<c007e9f4>] psr: 000f0093
[ 76.322138] sp : db201ba8 ip : c05a1788 fp : db201bdc
[ 76.333585] r10: 00000000 r9 : 00000005 r8 : c05a1640
[ 76.338801] r7 : c08d7000 r6 : c08d7400 r5 : c05a1778 r4 : 00000002
[ 76.345308] r3 : 00000020 r2 : 00000010 r1 : 00000000 r0 : c05a1788
[ 76.351814] Flags: nzcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
[ 76.359021] Control: 18c5387d Table: 1b1a804a DAC: 00000015
[ 76.364753] Process dd (pid: 1498, stack limit = 0xdb200238)
[ 76.370394] Stack: (0xdb201ba8 to 0xdb202000)
[ 76.374744] 1ba0: 00000010 00000020 010200da c05a1640 600f0013 c056f20c
[ 76.382891] 1bc0: 00561000 0000001b 010200da 00000000 db201c5c db201be0 c007f46c c007e93c
[ 76.391056] 1be0: 000000c1 db201bf0 c00d6030 c08d6ff4 00000018 0000001f 000000ba 00000000
[ 76.399222] 1c00: 00000000 00000000 00000000 c05a1660 ffffffff c0ad0228 c05a1d44 c0ad0224
[ 76.407369] 1c20: 00000001 c0ad0228 de1fa8c8 c05a1640 000f0021 010200da de1fa98c 00000000
[ 76.415534] 1c40: c05a1d40 00000000 00000000 c05a1640 db201ce4 db201c60 c007f9b4 c007f2f8
[ 76.423700] 1c60: 00000001 000000c1 c05a1640 00000002 de71ccc0 a00f0013 de1fa8c8 db200000
[ 76.431847] 1c80: db201c9c db201c90 d661f198 00000000 db201cc4 db201ca0 c0079864 c003a620
[ 76.440012] 1ca0: 00000002 00000001 c05a1640 de1fa98c ffffffff 00000000 db201ce4 00000000
[ 76.448177] 1cc0: de1fa98c 0000a705 ffffffff 00000000 0000000e 010200da db201d14 db201ce8
[ 76.456324] 1ce0: c007a308 c007f868 00001000 de1fa8c8 db200000 de1fa98c 00001000 00000000
[ 76.464490] 1d00: 0000000e de1fa98c db201d6c db201d18 c010a49c c007a2b0 00000000 0a705000
[ 76.472655] 1d20: 00001000 0000a705 0a706000 00000000 0a705000 00000000 c08d6c80 00000000
[ 76.480802] 1d40: 00000000 00000000 db200000 de1fa98c 00001000 00000000 00001000 c04097b4
[ 76.488968] 1d60: db201ddc db201d70 c00790c4 c010a42c 00001000 00000000 db201d98 db201d9c
[ 76.497133] 1d80: 00000000 00001000 ddd7b3c0 00020640 0a705000 00000000 c08d6c80 ddd7b3c0
[ 76.505280] 1da0: db201e98 00000001 00020640 000d3c00 526677a5 0a6e49c0 db201ed8 de1fa8c8
[ 76.513446] 1dc0: 00000000 00000000 ddd7b3c0 000f4240 db201e4c db201de0 c007ad90 c0078ff0
[ 76.521611] 1de0: 0a6e49c0 00000000 db201ed8 000f4240 00000000 00000001 db201e4c de1fa98c
[ 76.529758] 1e00: db201ed8 db201e98 0a6e49c0 00000000 db201ea0 00000001 00000000 000f4240
[ 76.537924] 1e20: db201e4c 0a6e49c0 00000000 de1fa930 00000001 de1fa8c8 ddd7b3c0 db201ea0
[ 76.546089] 1e40: db201e8c db201e50 c007ae74 c007aa0c 000f4240 00000000 db200000 db201e98
[ 76.554236] 1e60: c0576fc8 db201ea0 00000000 db201f78 ddd7b3c0 000f4240 db200000 db201ea0
[ 76.562402] 1e80: db201f44 db201e90 c00ae2b8 c007ae08 0a6e49c0 00000000 b6b94000 000f4240
[ 76.570567] 1ea0: 00000000 00000000 00000000 00000001 ffffffff ddd7b3c0 00000000 00000000
[ 76.578714] 1ec0: 00000000 00000000 de4a87c0 00000000 00000000 00000000 0a6e49c0 00000000
[ 76.586879] 1ee0: 00000000 00000000 000f4240 00000000 000f4240 00000000 00000000 00000000
[ 76.595045] 1f00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 76.603192] 1f20: 000f4240 ddd7b3c0 b6b94000 db201f78 b6b94000 000f4240 db201f74 db201f48
[ 76.611357] 1f40: c00ae9b0 c00ae230 00000000 00000000 bef2f4a8 0a6e49c0 00000000 ddd7b3c0
[ 76.619523] 1f60: 00000000 b6b94000 db201fa4 db201f78 c00aed28 c00ae908 0a6e49c0 00000000
[ 76.627670] 1f80: 00000000 000f4240 b6b94000 00000004 c000e2c4 00000000 00000000 db201fa8
[ 76.635835] 1fa0: c000e140 c00aecf0 00000000 000f4240 00000001 b6b94000 000f4240 00000000
[ 76.644001] 1fc0: 00000000 000f4240 b6b94000 00000004 00019438 000f4240 b6b94000 00019440
[ 76.652148] 1fe0: 00000000 bef2f49c 0000b745 b6ec625c 600f0010 00000001 00000000 00000000
[ 76.660295] Backtrace:
[ 76.662746] [<c007e930>] (__rmqueue+0x0/0x338) from [<c007f46c>] (get_page_from_freelist+0x180/0x570)
[ 76.671944] [<c007f2ec>] (get_page_from_freelist+0x0/0x570) from [<c007f9b4>] (__alloc_pages_nodemask+0x158/0x6f4)
[ 76.682284] [<c007f85c>] (__alloc_pages_nodemask+0x0/0x6f4) from [<c007a308>] (grab_cache_page_write_begin+0x64/0xb0)
[ 76.692864] [<c007a2a4>] (grab_cache_page_write_begin+0x0/0xb0) from [<c010a49c>] (ext3_write_begin+0x7c/0x204)
[ 76.702947] [<c010a420>] (ext3_write_begin+0x0/0x204) from [<c00790c4>] (generic_file_buffered_write+0xe0/0x250)
[ 76.713084] [<c0078fe4>] (generic_file_buffered_write+0x0/0x250) from [<c007ad90>] (__generic_file_aio_write+0x390/0x3fc)
[ 76.724033] [<c007aa00>] (__generic_file_aio_write+0x0/0x3fc) from [<c007ae74>] (generic_file_aio_write+0x78/0xe4)
[ 76.734355] [<c007adfc>] (generic_file_aio_write+0x0/0xe4) from [<c00ae2b8>] (do_sync_write+0x94/0xd4)
[ 76.743645] [<c00ae224>] (do_sync_write+0x0/0xd4) from [<c00ae9b0>] (vfs_write+0xb4/0x140)
[ 76.751884] [<c00ae8fc>] (vfs_write+0x0/0x140) from [<c00aed28>] (sys_write+0x44/0x74)
[ 76.759773] r8:b6b94000 r7:00000000 r6:ddd7b3c0 r5:00000000 r4:0a6e49c0
[ 76.766482] [<c00aece4>] (sys_write+0x0/0x74) from [<c000e140>] (ret_fast_syscall+0x0/0x30)
[ 76.774795] Code: e594200c e584501c e3720001 0a000000 (e7f001f2)
[ 76.780878] ---[ end trace 8e5ded5893b0f16b ]---
[ 76.785468] note: dd[1498] exited with preempt_count 1
Oct 22 13:03:34 cortex kernel: [ 76.287560] ------------[ cut here


 

 

Obviuously this should not happen and is not treated by any error-handling in the kernel. Does anyone have a clue, where this could be coming from?

 

 

I am still unsure wether to look at the Board (hardware) or at the linux kernel itself..

 

0 Kudos
5 Replies
Observer jcteiwes
Observer
10,547 Views
Registered: ‎01-23-2011

Re: linux-xlnx kernel oopses when doing heavy IO with SDHC on Zynq XC7Z030

Update: 

 

I found that this is related to the DDR3 Memory we have connected to our Zynq. As a workaround i made the design just use the lower half of the available address space and its running fine - never saw that error again.. 

 

tl;dr: still working on it 

0 Kudos
Scholar rfs613
Scholar
10,541 Views
Registered: ‎05-28-2013

Re: linux-xlnx kernel oopses when doing heavy IO with SDHC on Zynq XC7Z030

Interesting... have you tried running something like memtester (http://pyropus.ca/software/memtester/)?
0 Kudos
Observer jcteiwes
Observer
10,494 Views
Registered: ‎01-23-2011

Re: linux-xlnx kernel oopses when doing heavy IO with SDHC on Zynq XC7Z030

Hi,

 

I used memtester and also the bare-metal memtest which comes with the xilinx SDK. There is no sign of a physical error (so far).

Usually this bug occurs after around 130MB of data being written - when the caches are flushed after every 100MB of data the error does not occur..  So there seem to be at least some (mis)confiugration happening inside the linux kernel as well.

 

If someone is curious: fs-caches (dentries, inodes and pagecache) are flushed to disk by issuing 

 

echo 3 > /proc/sys/vm/drop_caches  

 

 

0 Kudos
Visitor julianwolff3
Visitor
1,838 Views
Registered: ‎11-04-2015

Re: linux-xlnx kernel oopses when doing heavy IO with SDHC on Zynq XC7Z030

Was there ever a solution to this problem. We are having the same problem on our board using e a XC7Z045.

memory test all good, writing to sd card using the above dd command yeilds something line 130 MB and then a stall

0 Kudos
Visitor julianwolff3
Visitor
1,808 Views
Registered: ‎11-04-2015

Re: linux-xlnx kernel oopses when doing heavy IO with SDHC on Zynq XC7Z030

The solution to the problem (at least in our case) was a misconfigured DDR3 RAM controller. On Address 0xf8006000

there is (among others) a configuration word which should read 0x81 for 32 bit access and 0x85 for 16 bit access.

we have two 16 bit memory units. Setting the value to 0x85 made the lower 256 MB memory appear again as the upper

256 MB memory like a mirror. Thus Linux crashed wheever we got over 256 MB memory. This also explains why using

just 256 MB always worked.

 

reg_ddrc_data_bus_width
3:2 rw 0x0

 

DDR bus width control
00: 32-bit
01: 16-bit

 

 

0 Kudos