cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
dlorych
Visitor
Visitor
265 Views
Registered: ‎09-14-2020

Xen: Kernel bug when assigning more memory to dom0

Hi,

I am running Xen and PetaLinux on a ZCU104 with 2 GB internal memory and 4 GB external DDR4. If i assign too much memory to dom0 such that it needs to allocate memory on the internal memory and the DDR4 (beginning at around 1800 MB), then i get a kernel bug. Also it always uses the 2-level ABI when running into the kernel bug, while it runs the FIFO-based ABI otherwise.

[    0.252118] xen:events: Using 2-level ABI
[    0.256306] Xen: initializing cpu0
[    0.259621] ------------[ cut here ]------------
[    0.264284] kernel BUG at arch/arm64/xen/../../arm/xen/enlighten.c:192!
[    0.270946] Internal error: Oops - BUG: 0 [#1] SMP
[    0.275789] Modules linked in:
[    0.278905] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.0-xilinx-v2019.2 #1
[    0.286168] Hardware name: ZynqMP ZCU104 RevC (DT)
[    0.291015] pstate: 60000005 (nZCv daif -PAN -UAO)
[    0.295864] pc : xen_starting_cpu+0x110/0x118
[    0.300270] lr : xen_starting_cpu+0xe8/0x118
[    0.304593] sp : ffffff800800bbc0
[    0.307966] x29: ffffff800800bbc0 x28: 0000000000000000
[    0.313329] x27: 0000000000000000 x26: ffffff80091d5170
[    0.318692] x25: ffffff80091d5358 x24: ffffffc4f3fc5558
[    0.324055] x23: 0000000000000000 x22: 0000000000000000
[    0.329418] x21: ffffff80091c88a0 x20: ffffff80091ae790
[    0.334781] x19: ffffff80091c8648 x18: 0000000000000010
[    0.340144] x17: 0000000000000000 x16: 0000000000000018
[    0.345507] x15: ffffffffffffffff x14: ffffff80091c8648
[    0.350870] x13: ffffff808927224f x12: ffffff8009272257
[    0.356233] x11: ffffff80091dc000 x10: ffffff800800b8d0
[    0.361596] x9 : 00000000ffffffd0 x8 : ffffff800858b170
[    0.366959] x7 : 0000000000000005 x6 : 0000000000000032
[    0.372322] x5 : ffffff8008c05240 x4 : 0000000000000000
[    0.377685] x3 : 0000000000000000 x2 : ffffff800800bc08
[    0.383048] x1 : 0000000000000000 x0 : ffffffffffffffea
[    0.388413] Process swapper/0 (pid: 1, stack limit = 0x(____ptrval____))
[    0.395158] Call trace:
[    0.397670]  xen_starting_cpu+0x110/0x118
[    0.401735]  cpuhp_invoke_callback+0x84/0x210
[    0.406146]  cpuhp_issue_call+0xe4/0x138
[    0.410125]  __cpuhp_setup_state_cpuslocked+0x120/0x2b8
[    0.415401]  __cpuhp_setup_state+0x4c/0x70
[    0.419554]  xen_guest_init+0x274/0x2a8
[    0.423446]  do_one_initcall+0x74/0x178
[    0.427339]  kernel_init_freeable+0xf0/0x24c
[    0.431664]  kernel_init+0x10/0xfc
[    0.435123]  ret_from_fork+0x10/0x18
[    0.438758] Code: f0006e20 f944d400 cb000300 17ffffed (d4210000)
[    0.444900] ---[ end trace 1f3c7ef6594e23de ]---
[    0.449590] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    0.449590]
[    0.458826] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    0.458826]  ]---

The external memory is working fine when running a system without Xen (and it also appears in 'xl info' when assigning less memory to dom0).

This is the source code of the enlighten.c: https://elixir.bootlin.com/linux/v4.19/source/arch/arm/xen/enlighten.c#L192

It seems like the call to HYPERVISOR_vcpu_op() returns an error, which is a hypercall, but it is hard to find out what exactly it does. The only documentation i could find is the following: https://xenbits.xen.org/docs/unstable/hypercall/arm/include,public,arch-arm.h.html

Also I am not sure how the ABI is connected to all of this. Apparently the 2-level ABI is a fallback to the FIFO-based ABI, but it is not clear how Xen chooses which ABI to execute or why the FIFO-based ABI could not be used.

Is anyone able to help me here? Thanks in advance

 

This is the complete boot sequence:

Xilinx Zynq MP First Stage Boot Loader
Release 2019.2   Sep 21 2020  -  07:22:46
Reset Mode      :       System Reset
Platform: Silicon (4.0), Cluster ID 0x80000000
Running on A53-0 (64-bit) Processor, Device Name: XCZU7EV
XFSBL_EEPROM PRESENT
Calc_Vadj is VADJ_1V8
XFSBL_ERROR_I2C_WRITE
FMC VADJ Configuration Not SuccessfulBoard Configuration successful
Processor Initialization Done
================= In Stage 2 ============
SD1 with level shifter Boot Mode
SD: rc= 0
File name is BOOT.BIN
Multiboot Reg : 0x0
Image Header Table Offset 0x8C0
*****Image Header Table Details********
Boot Gen Ver: 0x1020000
No of Partitions: 0x5
Partition Header Address: 0x440
Partition Present Device: 0x0
Initialization Success
======= In Stage 3, Partition No:1 =======
UnEncrypted data Length: 0x49AA7D
Data word offset: 0x49AA7D
Total Data word length: 0x49AA7D
Destination Load Address: 0xFFFFFFFF
Execution Address: 0x0
Data word offset: 0xFDB0
Partition Attributes: 0x26
Destination Device is PL, changing LoadAddress
Non authenticated Bitstream download to start now
DMA transfer done
PL Configuration done successfully
Partition 1 Load Success
======= In Stage 3, Partition No:2 =======
UnEncrypted data Length: 0x31DE
Data word offset: 0x31DE
Total Data word length: 0x31DE
Destination Load Address: 0xFFFEA000
Execution Address: 0xFFFEA000
Data word offset: 0x4AA830
Partition Attributes: 0x117
Partition 2 Load Success
======= In Stage 3, Partition No:3 =======
UnEncrypted data Length: 0x2BFED
Data word offset: 0x2BFED
Total Data word length: 0x2BFED
Destination Load Address: 0x10080000
Execution Address: 0x10080000
Data word offset: 0x4ADA10
Partition Attributes: 0x114
Partition 3 Load Success
======= In Stage 3, Partition No:4 =======
Skipping the Partition 0x4
                          Partition 4 Load Success
All Partitions Loaded 0(release):xilinx-v2019.1-12-g713dace9
NOTICE:  BL31: Built : 07:17:15, Sep 21 2020
PMUFW:  v1.1


U-Boot 2019.01 (Sep 21 2020 - 07:22:05 +0000)

Model: ZynqMP ZCU104 RevC
Board: Xilinx ZynqMP
DRAM:  6 GiB
EL Level:       EL2
Chip ID:        zu7ev
MMC:   mmc@ff170000: 0
Loading Environment from SPI Flash... SF: Detected n25q512a with page size 256 Bytes, erase size 64 KiB, total 64 MiB
OK
In:    serial@ff000000
Out:   serial@ff000000
Err:   serial@ff000000
Model: ZynqMP ZCU104 RevC
Board: Xilinx ZynqMP
Hit any key to stop autoboot:  0
ZynqMP> mmc dev $sdbootdev &&&& mmcinfo; load mmc $sdbootdev:$partid 0xC00000 boot.scr; source 0xC00000
switch to partitions #0, OK
mmc0 is current device
Device: mmc@ff170000
Manufacturer ID: 3
OEM: 5344
Name: SC16G
Bus Speed: 50000000
Mode : SD High Speed (50MHz)
Rd Block Len: 512
SD version 3.0
High Capacity: Yes
Capacity: 14.8 GiB
Bus Width: 4-bit
Erase Group Size: 512 Bytes
712 bytes read in 12 ms (57.6 KiB/s)
## Executing script at 00c00000
819536 bytes read in 70 ms (11.2 MiB/s)
18782720 bytes read in 1268 ms (14.1 MiB/s)
34127 bytes read in 18 ms (1.8 MiB/s)
libfdt fdt_add_subnode(): FDT_ERR_EXISTS
## Flattened Device Tree blob at 02200000
   Booting using the fdt blob at 0x2200000
   reserving fdt memory region: addr=2200000 size=a000
   Using Device Tree in place at 0000000002200000, end 000000000220cfff

Starting kernel ...

 Xen 4.11.2-pre
(XEN) Xen version 4.11.2-pre (SKYNETorych@skynet.internal) (aarch64-xilinx-linux-gcc (GCC) 8.2.0) debug=n  Mon Sep 21 07:28:39 UTC 2020
(XEN) Latest ChangeSet: Fri Aug 30 16:32:35 2019 -0700 git:0bb0d1c1f5-dirty
(XEN) Processor: 410fd034: "ARM Limited", variant: 0x0, part 0xd03, rev 0x4
(XEN) 64-bit Execution:
(XEN)   Processor Features: 0000000000002222 0000000000000000
(XEN)     Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
(XEN)     Extensions: FloatingPoint AdvancedSIMD
(XEN)   Debug Features: 0000000010305106 0000000000000000
(XEN)   Auxiliary Features: 0000000000000000 0000000000000000
(XEN)   Memory Model Features: 0000000000001122 0000000000000000
(XEN)   ISA Features:  0000000000011120 0000000000000000
(XEN) 32-bit Execution:
(XEN)   Processor Features: 00000131:00011011
(XEN)     Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
(XEN)     Extensions: GenericTimer Security
(XEN)   Debug Features: 03010066
(XEN)   Auxiliary Features: 00000000
(XEN)   Memory Model Features: 10201105 40000000 01260000 02102211
(XEN)  ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 100000 KHz
(XEN) GICv2 initialization:
(XEN)         gic_dist_addr=00000000f9010000
(XEN)         gic_cpu_addr=00000000f9020000
(XEN)         gic_hyp_addr=00000000f9040000
(XEN)         gic_vcpu_addr=00000000f9060000
(XEN)         gic_maintenance_irq=25
(XEN) GICv2: Adjusting CPU interface base to 0xf902f000
(XEN) GICv2: 192 lines, 4 cpus, secure (IID 0200143b).
(XEN) Using scheduler: null Scheduler (null)
(XEN) Initializing null scheduler
(XEN) WARNING: This is experimental software in development.
(XEN) Use at your own risk.
(XEN) Allocated console ring of 16 KiB.
(XEN) Bringing up CPU1
(XEN) Bringing up CPU2
(XEN) Bringing up CPU3
(XEN) Brought up 4 CPUs
(XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID
(XEN) P2M: 3 levels with order-1 root, VTCR 0x80023558
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading Dom00008001fff36000d kernel from boot module @ 0000000001000000
(XEN) Allocating 1:1 mappings totalling 1984MB for dom0:
(XEN) BANK[0] 0x00000008000000-0x00000078000000 (1792MB)
(XEN) BANK[1] 0x000004e8000000-0x000004f4000000 (192MB)
(XEN) Grant table range: 0x000004ffe00000-0x000004ffe40000
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Loading zImage from 0000000001000000 to 0000000008080000-0000000009269a00
(XEN) Loading dom0 DTB to 0x0000000010000000-0x0000000010007cab
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 340kB init memory.
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER4
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER8
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER12
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER16
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER20
(XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER0
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 4.19.0-xilinx-v2019.2 (oe-user@oe-host) (gcc version 8.2.0 (GCC)) #1 SMP Mon Sep 21 07:15:55 UTC 2020
[    0.000000] Machine model: ZynqMP ZCU104 RevC
[    0.000000] Xen 4.11 support found
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 256 MiB at 0x0000000068000000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] psci: SMC Calling Convention v1.1
[    0.000000] random: get_random_bytes called from start_kernel+0x94/0x3e4 with crng_init=0
[    0.000000] percpu: Embedded 23 pages/cpu @(____ptrval____) s54040 r8192 d31976 u94208
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: enabling workaround for ARM erratum 845719
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 499968
[    0.000000] Kernel command line: console=hvc0 earlycon=xen earlyprintk=xen root=/dev/mmcblk0p2
[    0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.000000] software IO TLB: mapped [mem 0x63fff000-0x67fff000] (64MB)
[    0.000000] Memory: 1632028K/2031616K available (11196K kernel code, 678K rwdata, 5608K rodata, 832K init, 323K bss, 137444K reserved, 262144K cma-reserved)
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=1.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] arch_timer: cp15 timer(s) running at 100.00MHz (virt).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x171024e7e0, max_idle_ns: 440795205315 ns
[    0.000003] sched_clock: 56 bits at 100MHz, resolution 10ns, wraps every 4398046511100ns
[    0.000216] Console: colour dummy device 80x25
[    0.204517] console [hvc0] enabled
[    0.207987] Calibrating delay loop (skipped), value calculated using timer frequency.. 200.00 BogoMIPS (lpj=400000)
[    0.218445] pid_max: default: 32768 minimum: 301
[    0.223220] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)
[    0.229875] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)
[    0.237915] ASID allocator initialised with 32768 entries
[    0.243041] xen:grant_table: Grant tables using version 1 layout
[    0.248567] Grant table initialized
[    0.252118] xen:events: Using 2-level ABI
[    0.256306] Xen: initializing cpu0
[    0.259621] ------------[ cut here ]------------
[    0.264284] kernel BUG at arch/arm64/xen/../../arm/xen/enlighten.c:192!
[    0.270946] Internal error: Oops - BUG: 0 [#1] SMP
[    0.275789] Modules linked in:
[    0.278905] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.0-xilinx-v2019.2 #1
[    0.286168] Hardware name: ZynqMP ZCU104 RevC (DT)
[    0.291015] pstate: 60000005 (nZCv daif -PAN -UAO)
[    0.295864] pc : xen_starting_cpu+0x110/0x118
[    0.300270] lr : xen_starting_cpu+0xe8/0x118
[    0.304593] sp : ffffff800800bbc0
[    0.307966] x29: ffffff800800bbc0 x28: 0000000000000000
[    0.313329] x27: 0000000000000000 x26: ffffff80091d5170
[    0.318692] x25: ffffff80091d5358 x24: ffffffc4f3fc5558
[    0.324055] x23: 0000000000000000 x22: 0000000000000000
[    0.329418] x21: ffffff80091c88a0 x20: ffffff80091ae790
[    0.334781] x19: ffffff80091c8648 x18: 0000000000000010
[    0.340144] x17: 0000000000000000 x16: 0000000000000018
[    0.345507] x15: ffffffffffffffff x14: ffffff80091c8648
[    0.350870] x13: ffffff808927224f x12: ffffff8009272257
[    0.356233] x11: ffffff80091dc000 x10: ffffff800800b8d0
[    0.361596] x9 : 00000000ffffffd0 x8 : ffffff800858b170
[    0.366959] x7 : 0000000000000005 x6 : 0000000000000032
[    0.372322] x5 : ffffff8008c05240 x4 : 0000000000000000
[    0.377685] x3 : 0000000000000000 x2 : ffffff800800bc08
[    0.383048] x1 : 0000000000000000 x0 : ffffffffffffffea
[    0.388413] Process swapper/0 (pid: 1, stack limit = 0x(____ptrval____))
[    0.395158] Call trace:
[    0.397670]  xen_starting_cpu+0x110/0x118
[    0.401735]  cpuhp_invoke_callback+0x84/0x210
[    0.406146]  cpuhp_issue_call+0xe4/0x138
[    0.410125]  __cpuhp_setup_state_cpuslocked+0x120/0x2b8
[    0.415401]  __cpuhp_setup_state+0x4c/0x70
[    0.419554]  xen_guest_init+0x274/0x2a8
[    0.423446]  do_one_initcall+0x74/0x178
[    0.427339]  kernel_init_freeable+0xf0/0x24c
[    0.431664]  kernel_init+0x10/0xfc
[    0.435123]  ret_from_fork+0x10/0x18
[    0.438758] Code: f0006e20 f944d400 cb000300 17ffffed (d4210000)
[    0.444900] ---[ end trace 1f3c7ef6594e23de ]---
[    0.449590] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    0.449590]
[    0.458826] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    0.458826]  ]---
Tags (2)
0 Kudos
0 Replies