Enabling ECC results in kernel panic during booting with initramfs

[joaragj]

I am trying to enable ECC for DDR on zynq-7000 (picozed 7030), by enabling ECC in vivado and setting the effective DRAM Bus Width to 16 Bit instead of 32.

As expected in Uboot the RAM size is now halfed due to the ECC overhead, however, when I now try to boot with u-boot the kernel panics.

One solution I found was that I could fool the bootloader by overwrite the memory node on the device tree during buiding with a ram size of 1G instead of 512MB the system will boot. This however causes a kernel panic if I use more than the 512MB ram that is actually available.

Anyone know how to solve this?

Here is the output from u-boot:

U-Boot 2019.01 (Feb 12 2020 - 14:03:41 +0000) Xilinx Zynq ZC702
 
CPU:   Zynq 7z030
Silicon: v3.1
DRAM:  ECC enabled 512 MiB
MMC:   mmc@e0100000: 0, mmc@e0101000: 1
Loading Environment from SPI Flash... SF: Detected n25q128 with page size 256 Bytes, erase size 64 KiB, total 16 MiB
*** Warning - bad CRC, using default environment
 
In:    serial@e0001000
Out:   serial@e0001000
Err:   serial@e0001000
Net:   ZYNQ GEM: e000b000, phyaddr ffffffff, interface rgmii-id
eth0: ethernet@e000b000
Hit any key to stop autoboot:  0
HYPSO-1 Booting. Current bootcount is 1 of 5
booting from SD
Error reading cluster
** Unable to read file image.ub **
## Loading kernel from FIT Image at 10000000 ...
Bad FIT kernel image format!
ERROR: can't get kernel image!
5980026 bytes read in 448 ms (12.7 MiB/s)
  design filename = "System_wrapper;UserID=0XFFFFFFFF;Version=2019.1"
  part number = "7z030sbg485"
  date = "2020/02/04"
  time = "16:50:09"
  bytes in bitstream = 5979916
zynq_align_dma_buffer: Align buffer at 1000006e to fffff80(swap 1)
INFO:post config was not run, please run manually if needed
toggling SD card
gpio: pin 54 (gpio 54) value is 1
booting from SD
Invalid FAT entry
16384 bytes read in 15000 ms (1000 Bytes/s)
Wrong Image Format for bootm command
ERROR: can't get kernel image!
Booting from eMMC
85652696 bytes read in 6214 ms (13.1 MiB/s)
## Loading kernel from FIT Image at 10000000 ...
   Using 'conf@system-top.dtb' configuration
   Verifying Hash Integrity ... OK
   Trying 'kernel@1' kernel subimage
     Description:  Linux kernel
     Type:         Kernel Image
     Compression:  uncompressed
     Data Start:   0x10000104
     Data Size:    4150432 Bytes = 4 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: 0x00008000
     Entry Point:  0x00008000
     Hash algo:    sha1
     Hash value:   7149873da4012249b560a681a838a74a65f1b579
   Verifying Hash Integrity ... sha1+ OK
## Loading ramdisk from FIT Image at 10000000 ...
   Using 'conf@system-top.dtb' configuration
   Verifying Hash Integrity ... OK
   Trying 'ramdisk@1' ramdisk subimage
     Description:  petalinux-user-image
     Type:         RAMDisk Image
     Compression:  gzip compressed
     Data Start:   0x103f8fa0
     Data Size:    81485739 Bytes = 77.7 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: unavailable
     Entry Point:  unavailable
     Hash algo:    sha1
     Hash value:   e841fc21f3c2841271fc6b14c569dd11d96be590
   Verifying Hash Integrity ... sha1+ OK
## Loading fdt from FIT Image at 10000000 ...
   Using 'conf@system-top.dtb' configuration
   Verifying Hash Integrity ... OK
   Trying 'fdt@system-top.dtb' fdt subimage
     Description:  Flattened Device Tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x103f56a4
     Data Size:    14394 Bytes = 14.1 KiB
     Architecture: ARM
     Hash algo:    sha1
     Hash value:   0b89fc0ef14d011a1929999ef7cd4ca9513046f4
   Verifying Hash Integrity ... sha1+ OK
   Booting using the fdt blob at 0x103f56a4
   Loading Kernel Image ... OK
   Loading Ramdisk to 0324a000, end 07ffffab ... OK
   Loading Device Tree to 03243000, end 03249839 ... OK
 
Starting kernel ...
 
Booting Linux on physical CPU 0x0
Linux version 4.19.0-xilinx-v2019.1 (oe-user@oe-host) (gcc version 8.2.0 (GCC)) #1 SMP PREEMPT Thu Feb 6 08:32:43 UTC 2020
CPU: ARMv7 Processor [413fc090] revision 0 (ARMv7), cr=18c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
OF: fdt: Machine model: xlnx,zynq-7000
bootconsole [earlycon0] enabled
Memory policy: Data cache writealloc
cma: Reserved 16 MiB at 0x1f000000
random: get_random_bytes called from start_kernel+0x80/0x3c4 with crng_init=0
percpu: Embedded 16 pages/cpu @(ptrval) s35916 r8192 d21428 u65536
Built 1 zonelists, mobility grouping on.  Total pages: 130048
Kernel command line: console=ttyPS0,115200 earlyprintk
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 151396K/524288K available (6144K kernel code, 205K rwdata, 1612K rodata, 1024K init, 132K bss, 356508K reserved, 16384K cma-reserved, 0K highmem)
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
    vmalloc : 0xe0800000 - 0xff800000   ( 496 MB)
    lowmem  : 0xc0000000 - 0xe0000000   ( 512 MB)
    pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
    modules : 0xbf000000 - 0xbfe00000   (  14 MB)
      .text : 0x(ptrval) - 0x(ptrval)   (7136 kB)
      .init : 0x(ptrval) - 0x(ptrval)   (1024 kB)
      .data : 0x(ptrval) - 0x(ptrval)   ( 205 kB)
       .bss : 0x(ptrval) - 0x(ptrval)   ( 133 kB)
rcu: Preemptible hierarchical RCU implementation.
rcu:    RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
    Tasks RCU enabled.
rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
efuse mapped to (ptrval)
slcr mapped to (ptrval)
L2C: platform modifies aux control register: 0x72360000 -> 0x72760000
L2C: DT/platform modifies aux control register: 0x72360000 -> 0x72760000
L2C-310 erratum 769419 enabled
L2C-310 enabling early BRESP for Cortex-A9
L2C-310 full line of zeros enabled for Cortex-A9
L2C-310 ID prefetch enabled, offset 1 lines
L2C-310 dynamic clock gating enabled, standby mode enabled
L2C-310 cache controller enabled, 8 ways, 512 kB
L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x76760001
zynq_clock_init: clkc starts at (ptrval)
Zynq clock init
sched_clock: 64 bits at 249MHz, resolution 4ns, wraps every 4398046511102ns
clocksource: arm_global_timer: mask: 0xffffffffffffffff max_cycles: 0x7350b89c29, max_idle_ns: 881590431910 ns
Switching to timer-based delay loop, resolution 4ns
clocksource: ttc_clocksource: mask: 0xffff max_cycles: 0xffff, max_idle_ns: 716713566 ns
timer #0 at (ptrval), irq=17
Console: colour dummy device 80x30
Calibrating delay loop (skipped), value calculated using timer frequency.. 499.99 BogoMIPS (lpj=2499999)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
CPU: Testing write buffer coherency: ok
CPU0: Spectre v2: using BPIALL workaround
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x100000 - 0x100060
rcu: Hierarchical SRCU implementation.
smp: Bringing up secondary CPUs ...
CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
CPU1: Spectre v2: using BPIALL workaround
smp: Brought up 1 node, 2 CPUs
SMP: Total of 2 processors activated (999.99 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
futex hash table entries: 512 (order: 3, 32768 bytes)
pinctrl core: initialized pinctrl subsystem
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
cpuidle: using governor menu
hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
hw-breakpoint: maximum watchpoint size is 4 bytes.
zynq-ocm f800c000.ocmc: ZYNQ OCM pool: 256 KiB @ 0x(ptrval)
zynq-pinctrl 700.pinctrl: zynq pinctrl initialized
console [ttyPS0] enabledat MMIO 0xe0001000 (irq = 26, base_baud = 3125000) is a xuartps
console [ttyPS0] enabled
bootconsole [earlycon0] disabled
bootconsole [earlycon0] disabled
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
media: Linux media interface: v0.10
videodev: Linux video capture interface: v2.00
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
PTP clock support registered
EDAC MC: Ver: 3.0.0
FPGA manager framework
Advanced Linux Sound Architecture Driver Initialized.
clocksource: Switched to clocksource arm_global_timer
NET: Registered protocol family 2
tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes)
TCP established hash table entries: 4096 (order: 2, 16384 bytes)
TCP bind hash table entries: 4096 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Trying to unpack rootfs image as initramfs...
rootfs image is not initramfs (write error); looks like an initrd
/initrd.image: incomplete write (74731520 != 81485739)
Freeing initrd memory: 79576K
hw perfevents: no interrupt-affinity property for /pmu@f8891000, guessing.
hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
workingset: timestamp_bits=30 max_order=16 bucket_order=0
jffs2: version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc.
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
io scheduler mq-deadline registered
io scheduler kyber registered
dma-pl330 f8003000.dmac: Loaded driver for PL330 DMAC-241330
dma-pl330 f8003000.dmac:    DBUFF-128x8bytes Num_Chans-8 Num_Peri-4 Num_Events-16
brd: module loaded
loop: module loaded
m25p80 spi0.0: found n25q128a13, expected n25q512a
m25p80 spi0.0: n25q128a13 (16384 Kbytes)
3 fixed-partitions partitions found on MTD device spi0.0
Creating 3 MTD partitions on "spi0.0":
0x000000000000-0x000000640000 : "boot"
0x000000640000-0x000000660000 : "bootenv"
0x000000660000-0x000001140000 : "kernel"
mtd: partition "kernel" extends beyond the end of device "spi0.0" -- size truncated to 0x9a0000
libphy: Fixed MDIO Bus: probed
CAN device driver interface
libphy: MACB_mii_bus: probed
Marvell 88E1510 e000b000.ethernet-ffffffff:00: attached PHY driver [Marvell 88E1510] (mii_bus:phy_addr=e000b000.ethernet-ffffffff:00, irq=POLL)
macb e000b000.ethernet eth0: Cadence GEM rev 0x00020118 at 0xe000b000 irq 28 (00:0a:35:00:1e:53)
e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-pci: EHCI PCI platform driver
usbcore: registered new interface driver cdc_acm
cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
usbcore: registered new interface driver usb-storage
i2c /dev entries driver
cdns-wdt f8005000.watchdog: Xilinx Watchdog Timer at (ptrval) with timeout 10s
EDAC MC0: Giving out device to module 1 controller synps_ddr_controller: DEV synps_edac (POLLED)
Xilinx Zynq CpuIdle Driver started
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
sdhci-pltfm: SDHCI platform and OF driver helper
mmc0: SDHCI controller on e0100000.mmc [e0100000.mmc] using ADMA
mmc1: SDHCI controller on e0101000.mmc [e0101000.mmc] using ADMA
ledtrig-cpu: registered to indicate activity on CPUs
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
fpga_manager fpga0: Xilinx Zynq FPGA Manager registered
NET: Registered protocol family 10
Segment Routing with IPv6
sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
NET: Registered protocol family 17
can: controller area network core (rev 20170425 abi 9)
NET: Registered protocol family 29
can: raw protocol (rev 20170425)
can: broadcast manager protocol (rev 20170425 t)
can: netlink gateway (rev 20170425) max_hops=1
Registering SWP/SWPB emulation handler
of-fpga-region fpga-full: FPGA Region probed
hctosys: unable to open rtc device (rtc0)
of_cfs_init
of_cfs_init: OK
ALSA device list:
  No soundcards found.
RAMDISK: gzip image found at block 0
RAMDISK: incomplete write (6620 != 16379)
write error
VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
Please append a correct "root=" boot option; here are the available partitions:
0100           16384 ram0
 (driver?)
0101           16384 ram1
 (driver?)
0102           16384 ram2
 (driver?)
0103           16384 ram3
 (driver?)
0104           16384 ram4
 (driver?)
0105           16384 ram5
 (driver?)
0106           16384 ram6
 (driver?)
0107           16384 ram7
 (driver?)
0108           16384 ram8
 (driver?)
0109           16384 ram9
 (driver?)
010a           16384 ram10
 (driver?)
010b           16384 ram11
 (driver?)
010c           16384 ram12
 (driver?)
010d           16384 ram13
 (driver?)
010e           16384 ram14
 (driver?)
010f           16384 ram15
 (driver?)
1f00            6400 mtdblock0
 (driver?)
1f01             128 mtdblock1
 (driver?)
1f02            9856 mtdblock2
 (driver?)
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
CPU0: stopping
CPU: 0 PID: 34 Comm: kworker/0:1 Not tainted 4.19.0-xilinx-v2019.1 #1
Hardware name: Xilinx Zynq Platform
Workqueue: events delayed_fput
[<c010f18c>] (unwind_backtrace) from [<c010af0c>] (show_stack+0x10/0x14)
[<c010af0c>] (show_stack) from [<c06872cc>] (dump_stack+0x80/0xa0)
[<c06872cc>] (dump_stack) from [<c010d794>] (ipi_cpu_stop+0x3c/0x70)
[<c010d794>] (ipi_cpu_stop) from [<c010dfc4>] (handle_IPI+0x6c/0x90)
[<c010dfc4>] (handle_IPI) from [<c032e234>] (gic_handle_irq+0x84/0x90)
[<c032e234>] (gic_handle_irq) from [<c0101a0c>] (__irq_svc+0x6c/0xa8)
Exception stack(0xde4edb98 to 0xde4edbe0)
db80:                                                       de4780c0 de4780c0
dba0: 00001000 c020f0bc 00000000 de4780c0 00000000 00000000 00007648 00000000
dbc0: 00000001 0000000c 00000020 de4edbe8 c03f63f8 c020f0bc a0000113 ffffffff
[<c0101a0c>] (__irq_svc) from [<c020f0bc>] (end_bio_bh_io_sync+0x0/0x48)
[<c020f0bc>] (end_bio_bh_io_sync) from [<00000000>] (  (null))
---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

[rfs613]

This seems that you are running out of memory while decompressing the ramdisk:

RAMDISK: incomplete write (6620 != 16379)
write error

Since the decompression failed, there is no inittamfs, and thus you end up with the "VFS failed to mount root FS" error.

I would try to reduce size of initramfs if possible, or boot into SD or UBI rootfs directly.

[joaragj]

The size of the ramdisk image is 77.7 MB, with 512 MB ram, size shouldn't really be a problem. To me It looks like the address U-boot loads the device tree to overlaps with where the ramdisk is loaded.

   Loading Kernel Image ... OK
   Loading Ramdisk to 0324a000, end 07ffffab ... OK
   Loading Device Tree to 03243000, end 03249839 ... OK

I believe it is something with how U-boot is configured by the petalinux sdk, but I have no idea how to fix it.

[joaragj]

Just bumping this issue up a bit. Have anyone in this forum successfully enabled ECC for DRAM with vivado and petalinux SDK?

[mnsgs]

Hi @joaragj ,

Some time ago I raised https://forums.xilinx.com/t5/Embedded-Linux/Zynq-7000-DDR-ECC-Verification-Using-Linux-4-14-EDAC-Driver/td-p/1020196 haven't received any feedback and issue is still unresolved.

[joaragj]

I still haven't been able to find a solution for my problem. As I said, the only way I can avoid kernel panic during booting is to overwrite the memory device tree node by pretending that the RAM is still 1 GB and and not 512 MB as it is after ECC is enabled. This however cause kernel panic at a later stage if  one of my applications running on linux consumes more than 512 MB ram. To me it looks like this has to be a bug with the petalinux SDK.

I am also willing to try alternative ECC solutions, such as the one described here: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842219/Zynq-7000+AP+SoC+-+32+Bit+DDR+Access+with+ECC+Tech+Tip

But unfortuently, this guide is not compatible with petalinux 2019.1, which I am using.

So my questions is, are there really no documented way of enabling ECC for DRAM for zynq-7000 devices with petalinux 2019.1 ?

thanks,

[joaragj · Accepted solution]

Found out the problem was that the allocated space for bootm was to small. (apparantly the size of this space decreased when I enabled ECC).

Solution was to add 

bootm_size=0x10000000 to u-boot extra env in project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h

and append

#undef CONFIG_SYS_BOOTMAPSZ

to the same file.