cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
airizar
Observer
Observer
12,106 Views
Registered: ‎01-29-2009

Problem Booting Linux (kernel panic)

Hi all,

 

I'm having problems booting Linux from my custom-board Zynq.

My board is mainly based on Zedboard but we introduced more DDR memory (1GB) and also substitute the QSPI with a NAND flash.

 

After booting my system with u-boot from the sdcard I get the following report:

--------------------------------------------------------

OEM: 3456                                                                                                                                               
Name: F0F0F                                                                                                                                             
Tran Speed: 50000000                                                                                                                                    
Rd Block Len: 512                                                                                                                                       
SD version 2.0                                                                                                                                          
High Capacity: Yes                                                                                                                                      
Capacity: 3.7 GiB                                                                                                                                       
Bus Width: 4-bit                                                                                                                                        
reading uEnv.txt                                                                                                                                        
310 bytes read in 7 ms (43 KiB/s)                                                                                                                       
Loaded environment from uEnv.txt                                                                                                                        
Importing environment from SD ...                                                                                                                       
Running uenvcmd ...                                                                                                                                     
Copying Linux from SD to RAM...                                                                                                                         
reading uImage                                                                                                                                          
2858976 bytes read in 393 ms (6.9 MiB/s)                                                                                                                
reading uImage-zedboard-zynq7-mmcblk0p2.dtb                                                                                                             
23080 bytes read in 16 ms (1.4 MiB/s)                                                                                                                   
## Booting kernel from Legacy Image at 00a00000 ...                                                                                                     
   Image Name:   Linux-3.8.11-xilinx                                                                                                                    
   Image Type:   ARM Linux Kernel Image (uncompressed)                                                                                                  
   Data Size:    2858912 Bytes = 2.7 MiB                                                                                                                
   Load Address: 00008000                                                                                                                               
   Entry Point:  00008000                                                                                                                               
   Verifying Checksum ... OK                                                                                                                            
## Flattened Device Tree blob at 00900000                                                                                                               
   Booting using the fdt blob at 0x00900000                                                                                                             
   Loading Kernel Image ... OK                                                                                                                          
OK                                                                                                                                                      
   Loading Device Tree to 1fff7000, end 1ffffa27 ... OK                                                                                                 
                                                                                                                                                        
Starting kernel ...                                                                                                                                     
                                                                                                                                                        
Booting Linux on physical CPU 0x0                                                                                                                       
Linux version 3.8.11-xilinx (aip@veronica) (gcc version 4.8.1 (GCC) ) #2 SMP PREEMPT Tue Oct 27 18:21:33 CET 2015                                       
CPU: ARMv7 Processor [413fc090] revision 0 (ARMv7), cr=18c5387d                                                                                         
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache                                                                                
Machine: Xilinx Zynq Platform, model: ZedBoard                                                                                                          
bootconsole [earlycon0] enabled                                                                                                                         
cma: CMA: reserved 16 MiB at 1ec00000                                                                                                                   
Memory policy: ECC disabled, Data cache writealloc                                                                                                      
PERCPU: Embedded 7 pages/cpu @c09a6000 s7616 r8192 d12864 u32768                                                                                        
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 130048                                                                             
Kernel command line: console=ttyPS0,115200 root=/dev/mmcblk0p2 rw rootwait earlyprintk                                                                  
PID hash table entries: 2048 (order: 1, 8192 bytes)                                                                                                     
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)                                                                                         
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)                                                                                          
__ex_table already sorted, skipping sort                                                                                                                
BUG: Bad page state in process swapper  pfn:01023                                                                                                       
page:c05be460 count:0 mapcount:-1073741824 mapping:  (null) index:0x0                                                                                   
page flags: 0x0()   Modules linked in:
[<c0013fb0>] (unwind_backtrace+0x0/0x11c) from [<c00766e0>] (bad_page+0xd8/0x104)
[<c00766e0>] (bad_page+0xd8/0x104) from [<c0076780>] (free_pages_prepare+0x74/0xfc)
[<c0076780>] (free_pages_prepare+0x74/0xfc) from [<c0077bf8>] (__free_pages+0x48/0x60)
[<c0077bf8>] (__free_pages+0x48/0x60) from [<c051ebdc>] (free_all_bootmem_core.part.10+0xcc/0x1cc)
[<c051ebdc>] (free_all_bootmem_core.part.10+0xcc/0x1cc) from [<c051ee00>] (free_all_bootmem+0x58/0x74)
[<c051ee00>] (free_all_bootmem+0x58/0x74) from [<c05158e0>] (mem_init+0xe0/0x5f4)
[<c05158e0>] (mem_init+0xe0/0x5f4) from [<c051064c>] (start_kernel+0x150/0x2ec)
[<c051064c>] (start_kernel+0x150/0x2ec) from [<00008078>] (0x8078)
Disabling lock debugging due to kernel taint                       

-----------------------------------------------------------------------------

 

After this it produces a lot of similar info regarding "BUG: Bad page state in process swapper"

Maybe relevant for resolving this issue is the below report from the kernel.

 

-----------------------------------------------------------------------------------------

Memory: 512MB = 512MB total                                                                                                                             
Memory: 489924k/497476k available, 26812k reserved, 0K highmem                                                                                          
Virtual kernel memory layout:                                                                                                                           
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)                                                                                                       
    fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)                                                                                                       
    vmalloc : 0xe0800000 - 0xff000000   ( 488 MB)                                                                                                       
    lowmem  : 0xc0000000 - 0xe0000000   ( 512 MB)                                                                                                       
    pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)                                                                                                       
    modules : 0xbf000000 - 0xbfe00000   (  14 MB)                                                                                                       
      .text : 0xc0008000 - 0xc050f8bc   (5151 kB)                                                                                                       
      .init : 0xc0510000 - 0xc0539dc0   ( 168 kB)                                                                                                       
      .data &colon; 0xc053a000 - 0xc0573688   ( 230 kB)                                                                                                       
       .bss : 0xc0573688 - 0xc059dcdc   ( 170 kB)                                                                                                       
Preemptible hierarchical RCU implementation.                                                                                                            
        Dump stacks of tasks blocking RCU-preempt GP.                                                                                                   
        RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.                                                                                            
NR_IRQS:16 nr_irqs:16 16                                                                                                                                
xslcr mapped to e0802000                                                                                                                                
Zynq clock init                     

------------------------------------------------------------------------

 

 

0 Kudos
11 Replies
linnj
Xilinx Employee
Xilinx Employee
12,098 Views
Registered: ‎09-10-2008

Hi,

 

Are you confident your DDR is working well as I would wonder about it? Many think that running a hello world or a simple memory test (not very exhaustive and not on all of memory) does the job while we find that Linux will expose memory issues.

 

Thanks

John

0 Kudos
airizar
Observer
Observer
12,094 Views
Registered: ‎01-29-2009

 

I've done memory test on the DDR without errors but you are right they are far from an exhaustive test.

I dit small tests at different memory locations without errors but I can go for a full test and see what happens.

Just running mtest in u-boot is an exhaustive test?

 

Thanks

 

Andoni

 

 

0 Kudos
linnj
Xilinx Employee
Xilinx Employee
12,084 Views
Registered: ‎09-10-2008

Hi Andoni,

 

I can't say on mtest as I've not used it. There is a standalone test in the Xilinx SDK I know. 

 

I noticed the amount of memory shows only 512 MB even though you said it has 1 GB. I assume this same kernel should run on a Zed board successfully just to ensure there's no s/w setup issues. 

 

Thanks

John

 

 

0 Kudos
airizar
Observer
Observer
12,072 Views
Registered: ‎01-29-2009

Thanks John

 

I'll try the standalone application and in the meantime will check if the images boot on the zedboard

 

Kind Regards


Andoni

 

0 Kudos
airizar
Observer
Observer
12,046 Views
Registered: ‎01-29-2009

I've made a thorough check of the 1GB DDR memory to see if there is any problem.

 

Using the standalone memory test, the DDR memory passed the test. 

 

Using the mtest command in u-boot I've found the following errors after checking all possible memory addresses:

zynq-uboot> mtest 0x3fB4AE10 0x3fB4AE20 0xAAAAAAAA 1 

Pattern AAAAAAAA Writing... Reading...
Mem error @ 0x3FB4AE10: found 3FFA4C60, expected AAAAAAAA
Mem error @ 0x3FB4AE14: found 3FF75D9C, expected AAAAAAAB
Mem error @ 0x3FB4AE18: found 00000000, expected AAAAAAAC
Mem error @ 0x3FB4AE1C: found AAAAAAAA, expected AAAAAAAD
Tested 1 iteration(s) with 4 errors.

 

When I boot the zynq with the memblock=debug bootarg I obtain the report below, without finally booting.

The report below keeps mentioning a 512MB memory space even though u-boot reports 1GB and is apparently properly addressing it, but the kernel seems unaware of the additional memory space. I initially thought that it was due to a wrong dts file, so I produced a new one with the SDK (2015.3) after exporting the Vivado project, compiled the kernel again, but no success.

Is there any way from the report below to find if any of the zynq devices is not properly memory mapped in Vivado? If so. where do I have to look at to find the offending device? Or is it the most probable cause a fault in the DDR?

 

Kind Regards

 

Andoni

 

 

Booting Linux on physical CPU 0x0
Linux version 3.8.11-xilinx (aip@veronica) (gcc version 4.8.1 (GCC) ) #3 SMP PREEMPT Tue Nov 3 11:09:07 CET 2015
CPU: ARMv7 Processor [413fc090] revision 0 (ARMv7), cr=18c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine: Xilinx Zynq Platform, model: ZedBoard
bootconsole [earlycon0] enabled
memblock_reserve: [0x00000000008200-0x000000005999dc] arm_memblock_init+0x4c/0x16c
memblock_reserve: [0x00000000004000-0x00000000008000] arm_memblock_init+0xf0/0x16c
memblock_reserve: [0x0000001fff7000-0x0000001ffffa28] arm_dt_memblock_reserve+0x24/0x84
memblock_reserve: [0x00000000000000-0x00000000004000] arm_memblock_init+0x104/0x16c
memblock_reserve: [0x0000001ec00000-0x0000001fc00000] memblock_alloc_base_nid+0x48/0x5c
cma: CMA: reserved 16 MiB at 1ec00000
MEMBLOCK configuration:
memory size = 0x20000000 reserved size = 0x15a2204
memory.cnt = 0x1
memory[0x0] [0x00000000000000-0x0000001fffffff], 0x20000000 bytes
reserved.cnt = 0x4
reserved[0x0] [0x00000000000000-0x00000000007fff], 0x8000 bytes
reserved[0x1] [0x00000000008200-0x000000005999db], 0x5917dc bytes
reserved[0x2] [0x0000001ec00000-0x0000001fbfffff], 0x1000000 bytes
reserved[0x3] [0x0000001fff7000-0x0000001ffffa27], 0x8a28 bytes
Memory policy: ECC disabled, Data cache writealloc
memblock_reserve: [0x0000001fffffe0-0x00000020000000] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001fff6000-0x0000001fff7000] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001fff5000-0x0000001fff6000] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001fff4000-0x0000001fff5000] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001fff3000-0x0000001fff4000] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001fff2000-0x0000001fff3000] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001fff1000-0x0000001fff2000] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001fff0000-0x0000001fff1000] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001ffef000-0x0000001fff0000] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001ffee000-0x0000001ffef000] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001ffed000-0x0000001ffee000] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001ffec000-0x0000001ffed000] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001fffffc0-0x0000001fffffe0] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001ffeb000-0x0000001ffec000] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001fffffa0-0x0000001fffffc0] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001ffea000-0x0000001ffeb000] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001ffe9000-0x0000001ffea000] memblock_alloc_base_nid+0x48/0x5c
memblock_reserve: [0x0000001ffe5000-0x0000001ffe9000] memblock_alloc_base_nid+0x48/0x5c
PERCPU: Embedded 7 pages/cpu @c09a2000 s7616 r8192 d12864 u32768
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048
Kernel command line: console=ttyPS0,115200 root=/dev/mmcblk0p2 rw rootwait earlyprintk memblock=debug
PID hash table entries: 2048 (order: 1, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
__ex_table already sorted, skipping sort
Memory: 512MB = 512MB total
Memory: 497492k/497492k available, 26796k reserved, 0K highmem
Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB)
vmalloc : 0xe0800000 - 0xff000000 ( 488 MB)
lowmem : 0xc0000000 - 0xe0000000 ( 512 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc050a6e4 (5130 kB)
.init : 0xc050b000 - 0xc0534dc0 ( 168 kB)
.data &colon; 0xc0536000 - 0xc056f448 ( 230 kB)
.bss : 0xc056f448 - 0xc05999dc ( 170 kB)
Preemptible hierarchical RCU implementation.
Dump stacks of tasks blocking RCU-preempt GP.
RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
NR_IRQS:16 nr_irqs:16 16
xslcr mapped to e0802000
Zynq clock init
sched_clock: 16 bits at 54kHz, resolution 18432ns, wraps every 1207ms
ps7-ttc #0 at e0804000, irq=43
Console: colour dummy device 80x30
Calibrating delay loop... 1332.01 BogoMIPS (lpj=6660096)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x3a7a10 - 0x3a7a44
L310 cache controller enabled
l2x0: 8 ways, CACHE_ID 0x000000c0, AUX_CTRL 0x72060000, Cache size: 524288 B
CPU1: Booted secondary processor
CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
Brought up 2 CPUs
SMP: Total of 2 processors activated (2664.03 BogoMIPS).
devtmpfs: initialized
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
xgpiops e000a000.ps7-gpio: gpio at 0xe000a000 mapped to 0xe080c000
hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
hw-breakpoint: maximum watchpoint size is 4 bytes.
bio: create slab <bio-0> at 0
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
PTP clock support registered
Switching to clocksource xttcps_clocksource
NET: Registered protocol family 2
TCP established hash table entries: 4096 (order: 3, 32768 bytes)
TCP bind hash table entries: 4096 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP: reno registered
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
hw perfevents: enabled with ARMv7 Cortex-A9 PMU driver, 7 counters available
jffs2: version 2.2. (NAND) (SUMMARY) �© 2001-2006 Red Hat, Inc.
msgmni has been set to 1003
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
dma-pl330 f8004000.ps7-dma: PERIPH_ID 0x0, PCELL_ID 0x0 !
dma-pl330: probe of f8004000.ps7-dma failed with error -22
dma-pl330 f8003000.ps7-dma: Loaded driver for PL330 DMAC-267056
dma-pl330 f8003000.ps7-dma: DBUFF-128x8bytes Num_Chans-8 Num_Peri-4 Num_Events-16
e0001��k˽���[ttyPS0] enabled, bootconsole disabled
console [ttyPS0] enabled, bootconsole disabled
xdevcfg f8007000.ps7-dev-cfg: ioremap f8007000 to e0864000 with size 100
brd: module loaded
loop: module loaded
xspips e0006000.ps7-spi: master is unqueued, this is deprecated
xspips e0006000.ps7-spi: at 0xE0006000 mapped to 0xE0866000, irq=58
xspips e0007000.ps7-spi: master is unqueued, this is deprecated
xspips e0007000.ps7-spi: at 0xE0007000 mapped to 0xE0868000, irq=81
xqspips e000d000.ps7-qspi: master is unqueued, this is deprecated

 

 

 

0 Kudos
rankeney
Participant
Participant
12,038 Views
Registered: ‎11-26-2014

FWIW, I was having problems with the amount of DDR being computed improperly (512MB instead of 1024MB), and the relocation of the kernel wasn't happening correctly. Turns out I was using Petalinux 2014.4 on a 32-bit CentOS system. When I tried 2015.2, I had to migrate to a 64-bit system and it then worked properly. Also trying 2014.4 on the 64-bit system gave the correct result. So make sure you're running on a 64-bit Linux system.

 

Robert

0 Kudos
airizar
Observer
Observer
12,026 Views
Registered: ‎01-29-2009

 

I'm using a 64-bit CentOS6 machine and the yocto distribution, not petalinux.

But thanks anyway for the tip.

 

Kind Regards

 

Andoni 

0 Kudos
jmadill
Visitor
Visitor
11,060 Views
Registered: ‎04-22-2015

My build is computing DDR size incorrectly too, same as your issue(512MB instead of 1024MB). I am building on Centos 64-bit with petalinux 2015.2.

 

Is there a setting somewhere that would cause this issue? Any help would be appreciated.

 

James

0 Kudos
airizar
Observer
Observer
11,053 Views
Registered: ‎01-29-2009

Hi James

 

In my case it was a problem with signal integrity in the DDR.

I managed to run the standalone DDR Test provided with the latest versions of Vivado. It is a very comprehensive test on DDR access and give you priceless information like eye diagrams, etc. 

 

I managed to solve the problem by reduced the DDR frequency from 533 MHz to 266 MHz and it worked.

We are now reviewing the PCB layout to find the source of the signal integrity problems

 

Regards

 

Andoni 

0 Kudos
jmadill
Visitor
Visitor
8,258 Views
Registered: ‎04-22-2015

Thanks for the response, 

 

In my case I am using the Trenz TE0715-03 board, so I dont think that signal integrity is my case. When I first booted the module with the supplied u-boot that was already installed from trenz u-boot see the 1GB ddr.

 

Once I created my own fpga design and built petalinux, u-boot now shows 512MB.

 

James

0 Kudos
airizar
Observer
Observer
8,254 Views
Registered: ‎01-29-2009

 

In my whereabouts trying to solve my problem I found that adding mem=1024M to the boot arguments in Linux solved the problem of letting know the kernel about the memory size.

 

Regards

 

Andoni 

0 Kudos