cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
mickh
Visitor
Visitor
721 Views
Registered: ‎07-21-2021

Petalinux 2020.2 kernel panic with INTC

Jump to solution

Hi there,

I'm trying to get petalinux 2020.2 up and running with a hardware design from Vivado 2020.2 on a z-turn board (http://www.myirtech.com/list.asp?id=502). I'm able to get a basic build running, and managed to add a couple of AXI Uartlite blocks and get those working and testing OK.  I handled their interrupts initially by running them through a Concat block and then into the IRQ_F2P on the Zynq PS.

Unfortunately, the end design will have too many interrupts to handle this way.  So my understanding after some reading here and elsewhere was to use the AXI Interrupt Controller block between the Concat and the IRQ_F2P, resulting in this design:

mickh_0-1627569864577.png

This validates in Vivado fine, and petalinux configures and builds ok using that as the hardware config.  But I get a kernel panic on boot every time.  The console shows this:

 

 

 

U-Boot 2020.01 (Jul 29 2021 - 14:29:15 +0000)

CPU:   Zynq 7z020
Silicon: v3.1
DRAM:  ECC disabled 1 GiB
Flash: 0 Bytes
NAND:  0 MiB
MMC:   mmc@e0100000: 0
Loading Environment from SPI Flash... SF: Detected w25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
*** Warning - bad CRC, using default environment

In:    serial@e0001000
Out:   serial@e0001000
Err:   serial@e0001000
Net:   
ZYNQ GEM: e000b000, mdio bus e000b000, phyaddr -1, interface rgmii-id

Warning: ethernet@e000b000 using MAC address from DT
eth0: ethernet@e000b000
Hit any key to stop autoboot:  2 ... 1 ... 0 
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
2010 bytes read in 17 ms (115.2 KiB/s)
## Executing script at 03000000
11562788 bytes read in 751 ms (14.7 MiB/s)
## Loading kernel from FIT Image at 10000000 ...
   Using 'conf@system-top.dtb' configuration
   Verifying Hash Integrity ... OK
   Trying 'kernel@1' kernel subimage
     Description:  Linux kernel
     Type:         Kernel Image
     Compression:  uncompressed
     Data Start:   0x100000e8
     Data Size:    4327424 Bytes = 4.1 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: 0x00200000
     Entry Point:  0x00200000
     Hash algo:    sha256
     Hash value:   fee4d913b5b40b19bf878dffd238364350ad26ad70a6fa89891fdb26b6db016c
   Verifying Hash Integrity ... sha256+ OK
## Loading ramdisk from FIT Image at 10000000 ...
   Using 'conf@system-top.dtb' configuration
   Verifying Hash Integrity ... OK
   Trying 'ramdisk@1' ramdisk subimage
     Description:  petalinux-image-minimal
     Type:         RAMDisk Image
     Compression:  uncompressed
     Data Start:   0x10425b10
     Data Size:    7212704 Bytes = 6.9 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: unavailable
     Entry Point:  unavailable
     Hash algo:    sha256
     Hash value:   35de6fe19c928bc678003bce4dd8945cc17680ced573a5b020e9399877f39304
   Verifying Hash Integrity ... sha256+ OK
## Loading fdt from FIT Image at 10000000 ...
   Using 'conf@system-top.dtb' configuration
   Verifying Hash Integrity ... OK
   Trying 'fdt@system-top.dtb' fdt subimage
     Description:  Flattened Device Tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x104209f4
     Data Size:    20558 Bytes = 20.1 KiB
     Architecture: ARM
     Hash algo:    sha256
     Hash value:   38d130413cabfab9d4ead715a55ee1a82bc8497786a3e3fd14d86c1fab6d7f36
   Verifying Hash Integrity ... sha256+ OK
   Booting using the fdt blob at 0x104209f4
   Loading Kernel Image
   Loading Ramdisk to 1f91f000, end 1ffffea0 ... OK
   Loading Device Tree to 1f916000, end 1f91e04d ... OK

Starting kernel ...

Booting Linux on physical CPU 0x0
Linux version 5.4.0-xilinx-v2020.2 (oe-user@oe-host) (gcc version 9.2.0 (GCC)) #1 SMP PREEMPT Wed Jul 28 03:13:40 UTC 2021
CPU: ARMv7 Processor [413fc090] revision 0 (ARMv7), cr=18c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
OF: fdt: Machine model: xlnx,zynq-7000
earlycon: cdns0 at MMIO 0xe0001000 (options '115200n8')
printk: bootconsole [cdns0] enabled
Memory policy: Data cache writealloc
cma: Reserved 16 MiB at 0x3f000000
percpu: Embedded 15 pages/cpu s31948 r8192 d21300 u61440
Built 1 zonelists, mobility grouping on.  Total pages: 260416
Kernel command line: console=ttyPS0,115200 earlycon root=/dev/ram0 rw
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
mem auto-init: stack:off, heap alloc:off, heap free:off
Memory: 1003844K/1048576K available (6144K kernel code, 217K rwdata, 1844K rodata, 1024K init, 131K bss, 28348K reserved, 16384K cma-reserved, 245760K highmem)
rcu: Preemptible hierarchical RCU implementation.
rcu: .RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
.Tasks RCU enabled.
rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
efuse mapped to (ptrval)
slcr mapped to (ptrval)
irq-xilinx: /amba_pl/interrupt-controller@41800000: num_irq=2, sw_irq=0, edge=0x3
L2C: platform modifies aux control register: 0x72360000 -> 0x72760000
L2C: DT/platform modifies aux control register: 0x72360000 -> 0x72760000
L2C-310 erratum 769419 enabled
L2C-310 enabling early BRESP for Cortex-A9
L2C-310 full line of zeros enabled for Cortex-A9
L2C-310 ID prefetch enabled, offset 1 lines
L2C-310 dynamic clock gating enabled, standby mode enabled
L2C-310 cache controller enabled, 8 ways, 512 kB
L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x76760001
random: get_random_bytes called from start_kernel+0x260/0x440 with crng_init=0
zynq_clock_init: clkc starts at (ptrval)
Zynq clock init
sched_clock: 64 bits at 249MHz, resolution 4ns, wraps every 4398046511102ns
clocksource: arm_global_timer: mask: 0xffffffffffffffff max_cycles: 0x7350b89c29, max_idle_ns: 881590431910 ns
Switching to timer-based delay loop, resolution 4ns
Console: colour dummy device 80x30
Calibrating delay loop (skipped), value calculated using timer frequency.. 499.99 BogoMIPS (lpj=2499999)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
CPU: Testing write buffer coherency: ok
CPU0: Spectre v2: using BPIALL workaround
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x100000 - 0x100060
rcu: Hierarchical SRCU implementation.
smp: Bringing up secondary CPUs ...
8<--- cut here ---
Unable to handle kernel NULL pointer dereference at virtual address 00000008
pgd = (ptrval)
[00000008] *pgd=00000000
Internal error: Oops - BUG: 805 [#1] PREEMPT SMP ARM
Modules linked in:
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.4.0-xilinx-v2020.2 #1
Hardware name: Xilinx Zynq Platform
PC is at xintc_write+0x58/0x80
LR is at arm_heavy_mb+0x18/0x4c
pc : [<c033fb50>]    lr : [<c01135d4>]    psr: 600001d3
sp : ef087f98  ip : 00000020  fp : 00000000
r10: 00000000  r9 : 413fc090  r8 : 0000406a
r7 : 00000000  r6 : 00000008  r5 : 00000000  r4 : ef6e306c
r3 : 00000000  r2 : 00000000  r1 : 00000000  r0 : f0808730
Flags: nZCv  IRQs off  FIQs off  Mode SVC_32  ISA ARM  Segment none
Control: 18c5387d  Table: 0000404a  DAC: 00000051
Process swapper/1 (pid: 0, stack limit = 0x(ptrval))
Stack: (0xef087f98 to 0xef088000)
7f80:                                                       ef6e306c 00000001
7fa0: 00000089 c033fd7c ef6dd294 00000001 00000089 00000000 0000406a c033fe3c
7fc0: ef6dd294 c011d0e0 00000000 00000047 00000002 c0b12f38 00000001 ef086000
7fe0: c0b36790 c010cf28 2f07c06a 00000051 10c0387d 0010242c 00000000 00000000
[<c033fb50>] (xintc_write) from [<c033fd7c>] (xil_intc_initial_setup+0x14/0xbc)
[<c033fd7c>] (xil_intc_initial_setup) from [<c033fe3c>] (xil_intc_start+0x18/0x28)
[<c033fe3c>] (xil_intc_start) from [<c011d0e0>] (notify_cpu_starting+0x6c/0x7c)
[<c011d0e0>] (notify_cpu_starting) from [<c010cf28>] (secondary_start_kernel+0xc4/0x11c)
[<c010cf28>] (secondary_start_kernel) from [<0010242c>] (0x10242c)
Code: ebf74e9d e5941000 e6bf5f35 e0816006 (e5865000) 
---[ end trace 0741a1527e4ee834 ]---
Kernel panic - not syncing: Attempted to kill the idle task!
CPU0: stopping
---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

 

 

 

I can see that something is going wrong in the initial setup of the INTC but I'm not clear exactly what's causing it.  This panic happens as soon as I add the INTC to the design - I tried getting rid of everything else and just having the Zynq PS and the INTC but got the same result.

I had to add the Uartlite drivers in petalinux kernel config but the interrupt controller driver was already selected.  Other than that it was by and large a default build.

Would be very grateful for any help working through this issue.  I have attached relevant device tree files and the tcl for the vivado project, along with the xsa file (some of them in a zip file because the website was complaining about mime type mismatches).  If there are other files that would help let me know and I will upload.

Thanks,

Mick

1 Solution

Accepted Solutions
basaro
Participant
Participant
490 Views
Registered: ‎01-26-2016

Hi Mick.

When Petalinux makes Device-tree auto-genarate , It is missed on Tool version.

Please correct as follows.

 

On Vivado.

Please open AXI INTC .

Chaange the 'Interrupt Output Connection' from BUS to Single.

Connect corectly AXI INTC irq and PS irq

make Bitstream.

export xsa files.

move to petalinux

petalinux-config --get-hw-descripion=<new xsa file>

petalinux-build

....

 

T.Koyama.

 

 

 

 

intc_chg.png

View solution in original post

11 Replies
maps-mpls
Mentor
Mentor
708 Views
Registered: ‎06-20-2017

Questions:

1.  Will you have more than 16 F2P interrupts?

2.  If so, your approach looks correct, though I have never tried it as I have never had a need.  Did you review and / or modify the device tree?  (For reference, compare your PL.dtsi to the following.  https://forums.xilinx.com/t5/Vitis-Acceleration-SDAccel-SDSoC/Vitis-platform-for-ultra96v2-is-not-working/m-p/1042130/highlight/true#M3804) noting that there may be some explainable changes due to tool evolution.

*** Destination: Rapid design and development cycles *** Please remember to give internet points to those who help you here. ***
mickh
Visitor
Visitor
700 Views
Registered: ‎07-21-2021

1. Yes, definitely more than 16

2. I did review the device tree - but I'm not very familiar with them so wasn't really sure what I was looking for.  The pl.dtsi is below.  I did at one point modify it to include an `interrupt-parent` line, which didn't work.  I did not include an `interrupts` line because I did not know how to calculate the values.  I do note that the `xlnx,kind-of-intr` value is different to that example you linked to, but I have configured the block as suggested in a few places (screenshot below) so not sure if that is significant.

mickh_0-1627580324597.png

 

/*
 * CAUTION: This file is automatically generated by Xilinx.
 * Version:  
 * Today is: Thu Jul 29 15:26:32 2021
 */


/ {
	amba_pl: amba_pl {
		#address-cells = <1>;
		#size-cells = <1>;
		compatible = "simple-bus";
		ranges ;
		axi_intc_0: interrupt-controller@41800000 {
			#interrupt-cells = <2>;
			clock-names = "s_axi_aclk";
			clocks = <&clkc 15>;
			compatible = "xlnx,axi-intc-4.1", "xlnx,xps-intc-1.00.a";
			interrupt-controller ;
			reg = <0x41800000 0x10000>;
			xlnx,kind-of-intr = <0x3>;
			xlnx,num-intr-inputs = <0x2>;
		};
		axi_uartlite_0: serial@42c00000 {
			clock-names = "s_axi_aclk";
			clocks = <&clkc 15>;
			compatible = "xlnx,axi-uartlite-2.0", "xlnx,xps-uartlite-1.00.a";
			current-speed = <115200>;
			device_type = "serial";
			interrupt-names = "interrupt";
			interrupt-parent = <&axi_intc_0>;
			interrupts = <1 0>;
			port-number = <1>;
			reg = <0x42c00000 0x10000>;
			xlnx,baudrate = <0x1c200>;
			xlnx,data-bits = <0x8>;
			xlnx,odd-parity = <0x0>;
			xlnx,s-axi-aclk-freq-hz-d = "100.0";
			xlnx,use-parity = <0x0>;
		};
		axi_uartlite_1: serial@42c10000 {
			clock-names = "s_axi_aclk";
			clocks = <&clkc 15>;
			compatible = "xlnx,axi-uartlite-2.0", "xlnx,xps-uartlite-1.00.a";
			current-speed = <115200>;
			device_type = "serial";
			interrupt-names = "interrupt";
			interrupt-parent = <&axi_intc_0>;
			interrupts = <0 0>;
			port-number = <2>;
			reg = <0x42c10000 0x10000>;
			xlnx,baudrate = <0x1c200>;
			xlnx,data-bits = <0x8>;
			xlnx,odd-parity = <0x0>;
			xlnx,s-axi-aclk-freq-hz-d = "100.0";
			xlnx,use-parity = <0x0>;
		};
	};
};

 

0 Kudos
maps-mpls
Mentor
Mentor
695 Views
Registered: ‎06-20-2017

Have you tried edge instead of level?  I know there once was a problem with edge but I think they fixed that.  Just grasping but hopefully it will be a quick experiment.

*** Destination: Rapid design and development cycles *** Please remember to give internet points to those who help you here. ***
patocarr
Teacher
Teacher
667 Views
Registered: ‎01-28-2008

Hi @mickh 

  I've had similar issues with INTC IP a while back and they turned out to be related to out-of-sync hardware vs Petalinux. The boot process hangs when the kernel tries to access the INTC IP, so it begs the question: is it present in the PL loaded with the kernel? For instance, the BOOT.BIN may have not been regenerated with the latest bitstream.

Thanks,

-Pat

 

Give kudos if helpful. Accept as solution if it solves your problem.
https://tuxengineering.com/blog

mickh
Visitor
Visitor
605 Views
Registered: ‎07-21-2021

Tried edge.  No change unfortunately

0 Kudos
mickh
Visitor
Visitor
598 Views
Registered: ‎07-21-2021

So, I think the hardware is in sync.  My process for changes is as follows:

  1. Re-generate bitstream in Vivado.  This generally forces saving changes to the design, and then re-running synthesis and implementation
  2. Export hardware in Vivado. Include the bitstream, and overwrite the previous xsa file.
  3. Run petalinux-config --get-hw-description=../z-turn_intc/design_1_wrapper.xsa. No options are changed, I just exit as soon as the menu appears which gives the following console output:

    INFO: Sourcing build tools
    INFO: Getting hardware description...
    INFO: Rename design_1_wrapper.xsa to system.xsa
    [INFO] Generating Kconfig for project
    [INFO] Menuconfig project


    *** End of the configuration.
    *** Execute 'make' to start the build or try 'make help'.

    [INFO] Sourcing build environment
    [INFO] Generating kconfig for Rootfs
    [INFO] Silentconfig rootfs
    [INFO] Generating plnxtool conf
    [INFO] Generating workspace directory

  4. Run petalinux-build
  5. Change to the images/linux directory
  6. Run petalinux-package --boot --fsbl --fpga system.bit --u-boot --dtb system.dtb --force to regenerate BOOT.BIN
  7. Copy the BOOT.BIN image.ub boot.scr files to the boot partition on the SD card

The system.bit is regenerated each petalinux build, so I guess then it is a question of whether the petalinux-config is distributing the xsa changes into the inputs to the build process.  I do not consistently run any kind of clean command before each build - occasionally I run petalinux-build -x mrproper if I have been messing around in some of the petalinux files and want to start from scratch.

Do you see any issues with that process?  Should I be cleaning out the build system each build?  Is there a way to verify that the system.bit matches the device tree?

Cheers,

Mick

0 Kudos
patocarr
Teacher
Teacher
593 Views
Registered: ‎01-28-2008

Hi @mickh 

  Your steps as listed make sense. There is a way to de-compile the device tree blob (dtb) into a human readable .dts with the following command:

$ cd ./images/linux
$ dtc -I dtb -O dts -o system.dts ./system.dtb 

  The output is the system.dts that will allow you to verify what hardware made it into the device tree. In my design, the AXI GPIO looks like this, where you can see the address range it sits at next to the <reg> property, i.e. 0xa0004000.

		gpio@a0004000 {
			#gpio-cells = <0x03>;
			clock-names = "s_axi_aclk";
			clocks = <0x19>;
			compatible = "xlnx,axi-gpio-2.0\0xlnx,xps-gpio-1.00.a";
			gpio-controller;
			reg = <0x00 0xa0004000 0x00 0x1000>;
			xlnx,all-inputs = <0x00>;
			xlnx,all-inputs-2 = <0x00>;
			xlnx,all-outputs = <0x00>;
			xlnx,all-outputs-2 = <0x00>;
			xlnx,dout-default = <0x00>;
			xlnx,dout-default-2 = <0x00>;
			xlnx,gpio-width = <0x0c>;
			xlnx,gpio2-width = <0x0c>;
			xlnx,interrupt-present = <0x00>;
			xlnx,is-dual = <0x01>;
			xlnx,tri-default = <0xffffffff>;
			xlnx,tri-default-2 = <0xffffffff>;
			phandle = <0x7c>;
		};

Thanks,

-Pat

Give kudos if helpful. Accept as solution if it solves your problem.
https://tuxengineering.com/blog

0 Kudos
mickh
Visitor
Visitor
569 Views
Registered: ‎07-21-2021

Hi Pat,

I have been decompiling the device tree for a while, and the changes do seem to be being reflected there.  What I don't know how to verify is whether that matches the bitstream.  I assumed the device tree and the bitstream were generated from the same source.  For example this is the section on the amba_pl:

        amba_pl {
                #address-cells = <0x01>;
                #size-cells = <0x01>;
                compatible = "simple-bus";
                ranges;
                phandle = <0x37>;

                interrupt-controller@41800000 {
                        #interrupt-cells = <0x02>;
                        clock-names = "s_axi_aclk";
                        clocks = <0x01 0x0f>;
                        compatible = "xlnx,axi-intc-4.1\0xlnx,xps-intc-1.00.a";
                        interrupt-controller;
                        reg = <0x41800000 0x10000>;
                        xlnx,kind-of-intr = <0x03>;
                        xlnx,num-intr-inputs = <0x02>;
                        phandle = <0x13>;
                };

                serial@42c00000 {
                        clock-names = "s_axi_aclk";
                        clocks = <0x01 0x0f>;
                        compatible = "xlnx,axi-uartlite-2.0\0xlnx,xps-uartlite-1.00.a";
                        current-speed = <0x1c200>;
                        device_type = "serial";
                        interrupt-names = "interrupt";
                        interrupt-parent = <0x13>;
                        interrupts = <0x01 0x00>;
                        port-number = <0x01>;
                        reg = <0x42c00000 0x10000>;
                        xlnx,baudrate = <0x1c200>;
                        xlnx,data-bits = <0x08>;
                        xlnx,odd-parity = <0x00>;
                        xlnx,s-axi-aclk-freq-hz-d = "100.0";
                        xlnx,use-parity = <0x00>;
                        phandle = <0x38>;
                };

                serial@42c10000 {
                        clock-names = "s_axi_aclk";
                        clocks = <0x01 0x0f>;
                        compatible = "xlnx,axi-uartlite-2.0\0xlnx,xps-uartlite-1.00.a";
                        current-speed = <0x1c200>;
                        device_type = "serial";
                        interrupt-names = "interrupt";
                        interrupt-parent = <0x13>;
                        interrupts = <0x00 0x00>;
                        port-number = <0x02>;
                        reg = <0x42c10000 0x10000>;
                        xlnx,baudrate = <0x1c200>;
                        xlnx,data-bits = <0x08>;
                        xlnx,odd-parity = <0x00>;
                        xlnx,s-axi-aclk-freq-hz-d = "100.0";
                        xlnx,use-parity = <0x00>;
                        phandle = <0x39>;
                };

As you can see, the uarts cite the INTC as their interrupt-parent, but the INTC does not have a parent (or interrupts for that matter) defined.  Do you know if it should?  The GIC is defined as follows:

interrupt-controller@f8f01000 {
                        compatible = "arm,cortex-a9-gic";
                        #interrupt-cells = <0x03>;
                        interrupt-controller;
                        reg = <0xf8f01000 0x1000 0xf8f00100 0x100>;
                        num_cpus = <0x02>;
                        num_interrupts = <0x60>;
                        phandle = <0x04>;
                };

There doesn't seem to be any connection between the INTC and the GIC, even though they are connected to each other as per the diagram in the original post.

Cheers,

Mick

0 Kudos
patocarr
Teacher
Teacher
555 Views
Registered: ‎01-28-2008

Hi Mickh

  It is indeed strange the GIC interrupt-controller is not listed as an interrupt-parent in the INTC node, but can't be sure it's causing the kernel crash. The Petalinux get-hw-description command should be importing the bitstream that comes in the .xsa and in the same file comes the actual hardware description (.hwh) with information about all hardware devices. The .xsa is merely a renamed zipped file. You could compare the .bit in the Vivado impl_1 directory with the one in project-spec/hw-description and in images/linux. They should all match.

Thanks,

-Pat

 

Give kudos if helpful. Accept as solution if it solves your problem.
https://tuxengineering.com/blog

0 Kudos
basaro
Participant
Participant
491 Views
Registered: ‎01-26-2016

Hi Mick.

When Petalinux makes Device-tree auto-genarate , It is missed on Tool version.

Please correct as follows.

 

On Vivado.

Please open AXI INTC .

Chaange the 'Interrupt Output Connection' from BUS to Single.

Connect corectly AXI INTC irq and PS irq

make Bitstream.

export xsa files.

move to petalinux

petalinux-config --get-hw-descripion=<new xsa file>

petalinux-build

....

 

T.Koyama.

 

 

 

 

intc_chg.png

View solution in original post

mickh
Visitor
Visitor
458 Views
Registered: ‎07-21-2021

Nailed it @basaro - that did the trick.  Many thanks for your help.

Thankyou also @patocarr and @maps-mpls for your suggestions - appreciate the help.