Petalinux 2020.1 vs. 2019.2 QSPI & JTAG boot

[patocarr]

Hi folks,

  I've been attempting to upgrade my custom hardware to Petalinux 2020.1 but nothing seems to work right, compared to 2019.2.

  QSPI programming.

  Using Vivado 2019.2 I can program the QSPI memory using hardware manager. No problem, as long as the boot mode pins are set to JTAG, the memory is programmed correctly every time.

  On Vivado 2020.1, I haven't been able to program the QSPI using the 2020.1 images. I've created a base Petalinux 2020.1 project with the following steps:

  $ petalinux-create -t project --template zynqMP -n DAQ16-Petalinux-2020.1-fresh
  $ cd DAQ16-Petalinux-2020.1-fresh
  $ petalinux-config --get-hw-description ../../Firmware/DDAQ4-Firmware/
  $ petalinux-build

  No options were set in the menuconfig, just close and save. The build completed without issues. Then packaged the BOOT.BIN as follows:

plnx@f1ba7eff299c:~/project/Petalinux/DAQ16-Petalinux-2020.1-fresh$ petalinux-package --boot --u-boot --fsbl --fpga --pmufw 
INFO: sourcing build tools
INFO: File in BOOT BIN: "/home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1-fresh/images/linux/zynqmp_fsbl.elf"
INFO: File in BOOT BIN: "/home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1-fresh/images/linux/pmufw.elf"
INFO: File in BOOT BIN: "/home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1-fresh/project-spec/hw-description/daq16_top.bit"
INFO: File in BOOT BIN: "/home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1-fresh/images/linux/bl31.elf"
INFO: File in BOOT BIN: "/home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1-fresh/images/linux/system.dtb"
INFO: File in BOOT BIN: "/home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1-fresh/images/linux/u-boot.elf"
INFO: Generating zynqmp binary package BOOT.BIN...


****** Xilinx Bootgen v2020.1
  **** Build date : May 26 2020-14:07:15
    ** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.


[INFO]   : Bootimage generated successfully

INFO: Binary is ready.

  Technically, programming this image into QSPI would work, however Vivado 2020.1 leaves these messages in the bash console:

Using default mini u-boot image file - /opt/Xilinx/Vivado/2020.1/data/xicom/cfgmem/uboot/zynqmp_qspi_x8_dual_parallel.bin
===== mrd->addr=0xFF5E0204, data=0x00000000 =====
BOOT_MODE REG = 0x0000
Downloading FSBL...
Running FSBL...
Finished running FSBL.

  The Tcl console shows these messages below, and failing to program the QSPI with a window showing "[Labtools 27-3161] Flash Programming Unsuccessful".

set_property PROGRAM.ADDRESS_RANGE  {use_file} [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.FILES [list "/home/pcarr/Work/Elk/DAQ/Petalinux/DAQ16-Petalinux-2020.1-fresh/images/linux/BOOT.BIN" ] [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.BIN_OFFSET {0} [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.ZYNQ_FSBL {/home/pcarr/Work/Elk/DAQ/Petalinux/DAQ16-Petalinux-2020.1-fresh/images/linux/zynqmp_fsbl.elf} [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.BLANK_CHECK  0 [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.ERASE  1 [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.CFG_PROGRAM  1 [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.VERIFY  0 [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.CHECKSUM  0 [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
startgroup 
program_hw_cfgmem -hw_cfgmem [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
Problem in running uboot
Flash programming initialization failed.
ERROR: [Labtools 27-3161] Flash Programming Unsuccessful
program_hw_cfgmem: Time (s): cpu = 00:00:00.23 ; elapsed = 00:00:11 . Memory (MB): peak = 9739.723 ; gain = 0.000 ; free physical = 2233 ; free virtual = 81117
ERROR: [Common 17-39] 'program_hw_cfgmem' failed due to earlier errors.

  However, if I select the Petalinux 2019.2 image BOOT.BIN and corresponding FSBL, the image is programmed correctly in Vivado 2020.1. This is a clue that something in the image (FSBL?) is not being set properly in the build.

  How can I narrow this down to find the culprit? What can be missing from the default 2020.1 FSBL?

  JTAG boot

  The 2020.1 petalinux-boot --jtag throws an error message, but the same options work in 2019.2. The hardware server is in a remote machine with the matching version as Petalinux. The error message follows:

$ petalinux-boot --jtag --u-boot --fpga --hw_server-url 192.168.1.177:3121
INFO: sourcing build tools
INFO: Use bitstream: "/home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1-fresh/images/linux/system.bit.
INFO: Please use --fpga --bitstream <BITSTREAM> to specify a bitstream if you want to use other bitstream.
INFO: Launching XSDB for file download and boot.
INFO: This may take a few minutes, depending on the size of your image.
rlwrap: warning: your $TERM is 'xterm' but rlwrap couldn't find it in the terminfo database. Expect some problems.: Function not implemented
INFO: Configuring the FPGA...                                                                   
INFO: Downloading bitstream: /home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1-fresh/images/linux/system.bit to the target.
INFO: Downloading ELF file: /home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1-fresh/images/linux/pmufw.elf to the target.
INFO: Downloading ELF file: /home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1-fresh/images/linux/zynqmp_fsbl.elf to the target.
Memory write error at 0xFFFC0000. MMU fault at VA 0xFFFC0000. Address translation fault. Fault status code 0x04
    invoked from within
"::tcf::eval -progress {
	    apply {{info} {
		if { $::xsdb::silent_mode == 0 } {
		    switch -- [lindex $info 0] {
			"info" {
			    puts -nonewline "\r[lindex $info 1]"
			}
			"warning" {
			    puts "\n[lindex $info 1]"
			}
			"data" {
			    puts -nonewline "\r[lindex $info 1] of [lindex $info 2] complete"
			}
			"done" {
			    puts "\n[lindex $info 1]"
			}
		    }
		    flush stdout
		}
		::xsdb::abort_check
	    }}} {::tcf::cache_enter tcfchan#0 {tcf_cache_eval {download_cache_client ::tcfclient#0::arg}}}"
    (procedure "::tcf::cache_eval_with_progress" line 2)
    invoked from within
"::tcf::cache_eval_with_progress $params(chan) [list download_cache_client $argvar] {
	    apply {{info} {
		if { $::xsdb::silent_mode == 0 } {
		    s..."
    (procedure "dow" line 99)
    invoked from within
"dow  "/home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1-fresh/images/linux/zynqmp_fsbl.elf""
    (file "/tmp/tmp.KSJGBsrwRv" line 22)
INFO: The XSDB log is as follows

  The bitstream is loaded as expected but again the FSBL is suspected to be crashing for some reason.

Thanks in advance for any insight.

-Pat

[jrhtech]

I'm just starting to figure this out myself but this post provides some info:

https://forums.xilinx.com/t5/ACAP-and-SoC-Boot-and/Programming-QSPI-flash-failing/m-p/1125527#M4983

jeff

[patocarr]

Thanks @jrhtech for your response and link.

Can we please get Xilinx to acknowledge the issue, or point us in the right direction?

At the moment in Petalinux 2020.1, QSPI programming is broken, likely due to some change in the FSBL code. This may also be causing issues with Petalinux JTAG boot.

Thanks again,

-Pat

[patocarr]

Hi folks,

  I can confirm that Vivado 2020.1 can program a Petalinux 2020.1 image if and only if it uses a FSBL 2019.2 to program the BOOT.BIN into QSPI. However, the BOOT.BIN includes the FSBL from 2020.1, so the FSBL 2019.2 is needed just to program the flash.

  Any ideas what's going on?

Thanks in advance,

-Pat

[stephenm]

Hello Pat,

I gave this a quick run on my end (I used the prebuilt images for the zcu102 BSP). I dont see any issue here:

Can you try the XSA used in the petalinux in Vitis with a simple hello world app (include the boot images), and test here to make sure this is OK?

[patocarr]

Hi @stephenm 

  Thanks for your response. FYI, I haven't been able to generate the test you suggested (i.e. hello world) and posted the issue in this post on missing software templates 

Thanks again,

-Pat

[patocarr]

Hi @stephenm 

  I was able to use Vitis 2020.1 and run a hello-world example app on my custom board, so proving that the Vitis-generated default FSBL is viable. However, the original issue regarding booting from Petalinux 2020.1 is still present (i.e. the MMU fault at VA 0x1100000 message).

  This points to a Petalinux configuration issue of some kind, but I'm at a loss how to narrow it down.

  I've also tried programming the QSPI using the Vitis-generated FSBL, but also failed as before. Vivado 2020.1 hardware manager prints out the following, indicating a u-boot problem:

set_property PROGRAM.ADDRESS_RANGE  {use_file} [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.FILES [list "/home/pcarr/Work/Elk/DAQ/Petalinux/DAQ16-Petalinux-2020.1/images/linux/BOOT.BIN" ] [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.BIN_OFFSET {0} [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.ZYNQ_FSBL {/home/pcarr/Work/Elk/DAQ/vitis/daq16_top/zynqmp_fsbl/executable.elf} [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.BLANK_CHECK  0 [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.ERASE  1 [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.CFG_PROGRAM  1 [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.VERIFY  0 [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
set_property PROGRAM.CHECKSUM  0 [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
startgroup 
program_hw_cfgmem -hw_cfgmem [ get_property PROGRAM.HW_CFGMEM [lindex [get_hw_devices xczu39dr_0] 0]]
Problem in running uboot
Flash programming initialization failed.
ERROR: [Labtools 27-3161] Flash Programming Unsuccessful

By the way, enabling u-boot autoconfig throws error messages. Not sure what it should be set to.

The message is not clear though:

ERROR: Task (/home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1/components/yocto/layers/meta-xilinx/meta-xilinx-bsp/recipes-bsp/u-boot/u-boot-xlnx_2020.1.bb:do_compile) failed with exit code '1'

The file it points to doesn't have the do_compile task, so it must be implied somehow:

plnx@65839ccb9bf8:~/project/Petalinux/DAQ16-Petalinux-2020.1$ cat /home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1/components/yocto/layers/meta-xilinx/meta-xilinx-bsp/recipes-bsp/u-boot/u-boot-xlnx_2020.1.bb
UBOOT_VERSION = "v2020.01"

UBRANCH ?= "xlnx_rebase_v2020.01"

SRCREV ?= "86c84c0d0f916ec00d5d76a32dc9372a25429ca9"

include u-boot-xlnx.inc
include u-boot-spl-zynq-init.inc

LICENSE = "GPLv2+"
LIC_FILES_CHKSUM = "file://README;beginline=1;endline=4;md5=744e7e3bb0c94b4b9f6b3db3bf893897"

# u-boot-xlnx has support for these
HAS_PLATFORM_INIT ?= " \
		xilinx_zynqmp_virt_config \
		xilinx_zynq_virt_defconfig \
		xilinx_versal_vc_p_a2197_revA_x_prc_01_revA \
		"

Any guidance will be appreciated.

Thanks,

-Pat

[stephenm]

It seems that's the tools are struggling to download the uboot elf into memory. Can you share the u-boot elf?

[patocarr]

Hi @stephenm 

  Please find the u-boot.elf attached (zipped to bypass filter).

  Here's some info from the file:

$ file images/linux/u-boot.elf 
images/linux/u-boot.elf: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped

$ readelf -a images/linux/u-boot.elf
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           AArch64
  Version:                           0x1
  Entry point address:               0x9000000
  Start of program headers:          64 (bytes into file)
  Start of section headers:          918496 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         1
  Size of section headers:           64 (bytes)
  Number of section headers:         5
  Section header string table index: 4

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .data             PROGBITS         0000000009000000  00010000
       00000000000d02de  0000000000000000  WA       0     0     1
  [ 2] .symtab           SYMTAB           0000000000000000  000e02e0
       0000000000000090  0000000000000018           3     2     8
  [ 3] .strtab           STRTAB           0000000000000000  000e0370
       0000000000000049  0000000000000000           0     0     1
  [ 4] .shstrtab         STRTAB           0000000000000000  000e03b9
       0000000000000021  0000000000000000           0     0     1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  p (processor specific)

There are no section groups in this file.

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000010000 0x0000000009000000 0x0000000009000000
                 0x00000000000d02de 0x00000000000d02de  RW     0x10000

 Section to Segment mapping:
  Segment Sections...
   00     .data 

There is no dynamic section in this file.

There are no relocations in this file.

The decoding of unwind sections for machine type AArch64 is not currently supported.

Symbol table '.symtab' contains 6 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND 
     1: 0000000009000000     0 SECTION LOCAL  DEFAULT    1 
     2: 00000000000d02de     0 NOTYPE  GLOBAL DEFAULT  ABS _binary_u_boot_bin_size
     3: 00000000090d02de     0 NOTYPE  GLOBAL DEFAULT    1 _binary_u_boot_bin_end
     4: 0000000009000000     0 NOTYPE  GLOBAL DEFAULT  ABS _start
     5: 0000000009000000     0 NOTYPE  GLOBAL DEFAULT    1 _binary_u_boot_bin_start

No version information found in this file.

[patocarr]

Hi folks,

  Following up on this issue, I followed the tutorial in ug1209 chapter 5, boot and configuration, in order to verify whether the Petalinux generated binaries were viable. I have followed the QSPI-boot mode using JTAG up to the point of actually burning the flash. It works fine, so there must be something from Petalinux that's hosing this process.

  So let's connect to the target using XSCT, and download the PMU firmware.

xsct% connect -url 192.168.1.177:3121                                                  
xsct% targets -set -filter {name =~ "PSU"}                                             
xsct% mwr 0xffca0038 0x1ff                                                             
xsct% targets                                                                          
  1  PS TAP
     2  PMU
       14  MicroBlaze PMU (Sleeping. No clock)
     3  PL
  5* PSU
     6  RPU (Reset)
        7  Cortex-R5 #0 (RPU Reset)
        8  Cortex-R5 #1 (RPU Reset)
     9  APU (L2 Cache Reset)
       10  Cortex-A53 #0 (APU Reset)
       11  Cortex-A53 #1 (APU Reset)
       12  Cortex-A53 #2 (APU Reset)
       13  Cortex-A53 #3 (APU Reset)
xsct% targets -set -filter {name =~ "MicroBlaze PMU"}                                  
xsct% dow {./images/linux/pmufw.elf}                                                   
Downloading Program -- /home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1/images/linux/pmufw.elf
	section, .vectors.reset: 0xffdc0000 - 0xffdc0007
	section, .vectors.sw_exception: 0xffdc0008 - 0xffdc000f
	section, .vectors.interrupt: 0xffdc0010 - 0xffdc0017
	section, .vectors.hw_exception: 0xffdc0020 - 0xffdc0027
	section, .text: 0xffdc0050 - 0xffdd0aeb
	section, .rodata: 0xffdd0aec - 0xffdd1e3b
	section, .data: 0xffdd1e3c - 0xffdd5ea7
	section, .sdata2: 0xffdd5ea8 - 0xffdd5ea7
	section, .sdata: 0xffdd5ea8 - 0xffdd5ea7
	section, .sbss: 0xffdd5ea8 - 0xffdd5ea7
	section, .bss: 0xffdd5ec0 - 0xffdd9beb
	section, .srdata: 0xffdd9bec - 0xffdda507
	section, .stack: 0xffdda508 - 0xffddb507
	section, .xpbr_serv_ext_tbl: 0xffddf6e0 - 0xffddfadf
100%    0MB   0.2MB/s  00:00                                                           
Setting PC to Program Start Address 0xffdd049c
Successfully downloaded /home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1/images/linux/pmufw.elf
xsct% Info: MicroBlaze PMU (target 14) Stopped at 0xffd02f24 (Stop)
xsct% con                                                                              
Info: MicroBlaze PMU (target 14) Running (Sleeping. No clock)

  Then, let's target A53#0 and download the FSBL:

xsct% targets -set -filter {name =~ "Cortex-A53 #0"}                                   
xsct% rst -processor                                                                   
WARNING: If the reset is being triggered after powering on the device,
         write bootloop at reset vector address (0xffff0000), or use
         -clear-registers option, to avoid unpredictable behavior.
         Further warnings will be suppressed
xsct% Info: Cortex-A53 #0 (target 10) Stopped at 0xffff0000 (Reset Catch)              
xsct% dow {./images/linux/zynqmp_fsbl.elf}                                             
Downloading Program -- /home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1/images/linux/zynqmp_fsbl.elf
	section, .text: 0xfffc0000 - 0xfffd1aff
	section, .note.gnu.build-id: 0xfffd1b00 - 0xfffd1b23
	section, .init: 0xfffd1b40 - 0xfffd1b73
	section, .fini: 0xfffd1b80 - 0xfffd1bb3
	section, .rodata: 0xfffd1bc0 - 0xfffd2117
	section, .sys_cfg_data: 0xfffd2140 - 0xfffd2987
	section, .mmu_tbl0: 0xfffd3000 - 0xfffd300f
	section, .mmu_tbl1: 0xfffd4000 - 0xfffd5fff
	section, .mmu_tbl2: 0xfffd6000 - 0xfffd9fff
	section, .data: 0xfffda000 - 0xfffdb1f7
	section, .sbss: 0xfffdb1f8 - 0xfffdb1ff
	section, .bss: 0xfffdb200 - 0xfffdd6ff
	section, .heap: 0xfffdd700 - 0xfffddaff
	section, .stack: 0xfffddb00 - 0xfffdfaff
	section, .dup_data: 0xfffdfb00 - 0xfffe0cf7
	section, .handoff_params: 0xfffe9e00 - 0xfffe9e87
	section, .bitstream_buffer: 0xffff0040 - 0xfffffc3f
100%    0MB   0.2MB/s  00:00                                                           
Setting PC to Program Start Address 0xfffc0000
Successfully downloaded /home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1/images/linux/zynqmp_fsbl.elf
xsct% con                                                                              
Info: Cortex-A53 #0 (target 10) Running

  At this point, the tutorial states the FSBL should print its messages, but so far there has been no output to the console, i.e. no PMU or FSBL. The message does print out as soon as I stop the processor. Perhaps there's some buffering going on and not really a problem.

  Let's continue with ATF, after stopping the FSBL:

xsct% stop                                                                             
Info: Cortex-A53 #0 (target 10) Stopped at 0xfffd0138 (External Debug Request)         
xsct% dow {./images/linux/bl31.elf}                                                    
Downloading Program -- /home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1/images/linux/bl31.elf
	section, .text: 0xfffea000 - 0xffff1fff
	section, .rodata: 0xffff2000 - 0xffff2fff
	section, .data: 0xffff3000 - 0xffff67af
	section, stacks: 0xffff67c0 - 0xffff78bf
	section, .bss: 0xffff78c0 - 0xffff865f
	section, xlat_table: 0xffff9000 - 0xffffdfff
	section, coherent_ram: 0xffffe000 - 0xffffefff
100%    0MB   0.2MB/s  00:00                                                           
Setting PC to Program Start Address 0xfffea000
Successfully downloaded /home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1/images/linux/bl31.elf
xsct% con 
Info: Cortex-A53 #0 (target 10) Running
xsct% stop 
Info: Cortex-A53 #0 (target 10) Stopped at 0x26c611d36d12f200 (External Debug Request)

  The ATF outputs its message to the console as expected.

  Now let's load u-boot and run it.

xsct% dow {./images/linux/u-boot.elf}                                                  
Downloading Program -- /home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1/images/linux/u-boot.elf
	section, .data: 0x08000000 - 0x080e1f4d
100%    0MB   0.2MB/s  00:04                                                           
Setting PC to Program Start Address 0x08000000
Successfully downloaded /home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1/images/linux/u-boot.elf
xsct% con                                                                              
Info: Cortex-A53 #0 (target 10) Running
xsct% stop                                                                             
Info: Cortex-A53 #0 (target 10) Stopped at 0x7fe34298 (External Debug Request)

  As expected, u-boot ran and it was stopped at the console before booting the kernel, as it hasn't been loaded yet.

 

  Now let's load the BOOT.BIN and attempt to program QSPI over u-boot:

xsct% dow -data {./images/linux/BOOT.BIN} 0x2000000                                    
100%   34MB   0.3MB/s  01:45                                                           
Successfully downloaded /home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1/images/linux/BOOT.BIN
xsct% con                                                                              
Info: Cortex-A53 #0 (target 10) Running                                                

  Then in the target console, in u-boot we erase and program the QSPI:

 To note, the BOOT.BIN doesn't include the kernel (image.ub), so we'll load that now.

xsct% dow -data {./images/linux/image.ub} 0x2000000                                    
100%   45MB   0.3MB/s  02:21                                                           
Successfully downloaded /home/plnx/project/Petalinux/DAQ16-Petalinux-2020.1/images/linux/image.ub

  Then we flash the QSPI with the kernel image:

 All this shows that the images are apparently fine and can be loaded and executed from the XSCT command line tool. The question continues to be, why do petalinux-boot or Vivado hardware manager not work? How else can I verify what's wrong?

Thanks again for any insight,

-Pat

[patocarr]

Hi @stephenm 

  There's something suspicious about loading the FSBL using XSCT. The elf binary loads fine, but takes about 30 seconds of runtime to output its message to the screen. This is new in 2020.1, never took that long before. What could be delaying it? This may be a clue as to why petalinux-boot and Vivado can't perform as expected: they require the FSBL to be running without delay.

Will appreciate any insight.

Thanks,

-Pat

[patocarr]

Hi folks,

  Continuing on this path to a successful Petalinux 2020.1 boot... still no joy.

  I have generated a .tcl file from petalinux-boot to inspect the xsct instructions that are used to JTAG boot the device:

$ petalinux-boot --jtag --u-boot --hw_server-url 192.168.1.177:3121 --tcl boot-fpga.tcl

  Running the script with xsct fails in the same way as the petalinux-boot command. However, if I modify the script (attached) to add a 40 sec pause after running the FSBL, the script completes without error(!).

  Yet another indication the FSBL is stalling for a long time without [a known] reason.

Any ideas how to debug and proceed from here?

Thanks again,

-Pat

[stephenm]

Can you enable the debug in the fsbl?

[patocarr]

Hi @stephenm 

  Thanks for your suggestion. I've generated a ZynqMP FSBL in Vitis using FSBL_DEBUG_DETAILED flag and added it to the boot-fpga.tcl script. It doesn't print anything until some 30 seconds have elapsed, then the output looks normal.

 What could be the holdup in running FSBL?

Thanks,

-Pat

[patocarr]

Hi @stephenm 

  I've made some progress on this case, using a process of elimination. I started with the base PCM configuration, with only a GPIO IP on the PL. This much smaller design _also_ had the same 30 second delay before FSBL would print anything. So I went further and disabled the PCIe and SATA from the PCM.

  This worked. The 30-sec delay was gone. Just to be sure, I re-enabled PCIe, and the delay came back. Removed it again, and FSBL blazed through. So it's definitely related to PCIe. To note, PCIe is set as endpoint and there's no PCIe connected on the board; it's cabled PCIe and it's just not hooked up to a host, i.e. no clock.

  Looking at the FSBL code, there's a couple PCIe configuration sections for ZCU102, otherwise there's no active code related to PCIe.

  Comparing to the build in Petalinux 2019.2, that works fine, I notice the u-boot config target is set to xilinx_zynqmp_zcu102_rev1_0_defconfig. So I assume FSBL is running with XPS_BOARD_ZCU102 being set, and thus is running those PCIe-related blocks, and it's probably the reason this didn't happen in 2019.2.

  However, in Petalinux 2020.1 that ZCU102 defconfig does not exist, so the build fails. How do you suggest would be the best way to handle this?

a) Setting XPS_BOARD_ZCU102 in some config script under meta-user. Not sure where.

b) Adding an FSBL hook. Which one?

c) Setting some other config setting in meta-user.

d) Some other way of setting the ZCU102 defconfig I'm not aware of.

Thanks in advance for any insight.

-Pat

[patocarr]

Hi folks

  Here's some more info about this weird pause before FSBL is loaded and it starts running. It only happens when the PS PCIe is enabled, either root or endpoint mode. I can replicate it using Vitis and a hello world example, not just in Petalinux 2020.1. The difference is that Vitis waits for some feedback from FSBL (i.e. a breakpoint) and if it doesn't get it after 60 seconds, it times out. Petalinux just blindly waits 4 seconds and moves ahead with the next binary to load (i.e. system.dtb) regardless of the state of FSBL, throwing the "MMU fault" shown in a previous post if FSBL hasn't run due to this delay. In a similar way, Vivado also blindly tries to burn the flash assuming FSBL has run, which explains the QSPI flash error.

  I've tried patching the GT settings for ZCU102 in FSBL but didn't make any difference. It seems like FSBL _does not run_ until this initial ~30 second pause after loading it and running it.

  What else could be halting execution this long?? Why does it happen when PS PCIe is enabled, and goes away when disabled?

Thanks for any insight,

-Pat

[Edit: added info on Petalinux load process and Vivado case]

[patocarr]

Hi folks,

  I've been stuck on this issue for almost a month, and need some guidance to proceed.

  To summarize, the FSBL is apparently taking about 30 seconds to output anything to the console, and this happens when PS PCIe or DisplayPort are enabled in the PSU. This leads me to believe there's some kind of transceiver locked delay of sorts. However, this was not the case with 2019.2 or earlier.

  I can not reproduce this issue using a ZCU102 using no-board flow and no BSP, so there's a concern this custom hardware is somehow triggering this. The minimal design has only a PS configured just to be able to run Linux.

  I fail to see anything relevant in the FSBL that would hold printing the banner for this long and much less related to PS GTR.

 I will appreciate any ideas how to keep debugging this issue.

Thanks in advance,

-Pat

[shantmoses]

Hi @patocarr ,

I ran into a similar error with the JTAG boot in petalinux 2020.2 for a Gen3 RFSoC custom board with a PS PCIe end point while trying to boot up in a stand-alone mode (i.e not plugged into a PCIe slot). I was able to narrow down on the root cause of the 17 seconds delay in executing the FSBL code.

Basically in 2020.2 (or maybe in 2020.1), a function called "serdes_illcalib" has been added to the psu_init.c code that calls another function "serdes_illcalib_pcie_gen1" twice in case of Gen2 enabled PCIe link to run PCIe dynamic search on all active PCIe lanes and reset sequence. It is this "serdes_illcalib_pcie_gen1" function that gets into a loop to initialize and check the PCIe link status for up to 64 iterations before timing out in case there is no PCIe root port to link with. This loop of 64 iterations takes ~8 seconds to run. Therefore, manually running the commands in the JTAG boot tcl dump script, shows the FSBL debug prints after ~17 seconds from the time when the zynqmp_fsbl.elf is executed.

This feels problematic as we will need to maintain two versions of the logic and the petalinux build for the board; one with PS PCIe enabled and the other without PS PCIe, to run the board in a PC or stand-alone mode respectively.

I wonder if there is a workaround to this issue without having to remove the PS PCIe link from the logic!!!

Thanks,

Shant

[shantmoses]

Hi @patocarr ,

I found another FSBL process not given enough time by the petalinux JTAG boot command. The PS DDR ECC initialization for the upper PS DDR region doesn't finish before the boot script stops the FSBL code execution. After commenting out the second call to the "XFsbl_EccInit" function in the xfsbl_initialization.c file, compiling the FSBL in Vitis, and copying this hacked zynqmp_fsbl.elf into the petalinux project /images/linux/ directory, I was able to issue the following JTAG boot command "petalinux-boot --jtag --u-boot --fpga" and get the system to boot successfully into u-boot.

Programming the QSPI flash in Vivado used to fail but this hacked FSBL seems to fix that problem as well.

Apparently changes introduced into the FSBL source code after 2019.2 were not well tested and verified by Xilinx and broke a lot of functionality that used to work in the past.

Regards,

Shant