cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
patocarr
Teacher
Teacher
684 Views
Registered: ‎01-28-2008

Petalinux won't boot over JTAG

Jump to solution

Hi folks,

  I'm chasing an issue with this custom RFSoC-based board that somehow doesn't boot over JTAG, but boots and works fine from QSPI. I'm positive this JTAG boot has worked in the past year, so the hardware seems to be fine. I'm now using Vivado and Petalinux 2020.2.

  The QSPI memory is programmed over JTAG using either Vivado hardware manager or from the program_flash script from Vitis. The board also has an eMMC device that works properly when mounted from Linux, but using it as a boot device does not work either. I think these two issues, booting from JTAG and programming the eMMC may be related, as I show below.

  Booting Petalinux from JTAG and programming the eMMC over JTAG, both use the ATF binary (BL31) and it seems to be that on both cases, the target apparently dies after ATF runs. At least that's the last output that's seen.

  Looking for some clues how to keep debugging this issue. Any help is appreciated.

Thanks
-Pat

 

Petalinux boot from JTAG log, host terminal output:

[pcarr@storm linux]$ petalinux-boot --jtag --hw_server-url 192.168.1.177:3121 --u-boot
INFO: Sourcing build tools
INFO: Launching XSDB for file download and boot.
INFO: This may take a few minutes, depending on the size of your image.
rlwrap: warning: your $TERM is 'xterm-256color' but rlwrap couldn't find it in the terminfo database. Expect some problems.: Inappropriate ioctl for device
INFO: Downloading ELF file: ~/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/pmufw.elf to the target.
INFO: Downloading ELF file: ~/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/zynqmp_fsbl.elf to the target.
INFO: Loading image: ~/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/system.dtb at 0x01100000
INFO: Downloading ELF file: ~/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/u-boot.elf to the target.
INFO: Downloading ELF file: ~/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/bl31.elf to the target.
<No further output, as expected>

 

Petalinux boot from JTAG log, UART output:

Xilinx Zynq MP First Stage Boot Loader
Release 2020.2   Mar 26 2021  -  20:36:24
Reset Mode      :       System Reset
Platform: Silicon (4.0), Cluster ID 0x80000000
Running on A53-0 (64-bit) Processor, Device Name: XCZUUNKNEG
Processor Initialization Done
================= In Stage 2 ============
In JTAG Boot Mode
================= In Stage 4 ============
PM Init Success
Protection configuration applied
Exit from FSBL
NOTICE:  ATF running on XCZU39DR/silicon v4/RTL5.1 at 0xfffea000
NOTICE:  BL31: v2.2(release):xilinx_rebase_v2.2_2020.2
NOTICE:  BL31: Built : 21:21:06, Apr  7 2021
<No further output; expected U-Boot to run>

 

Running xsct after this boot shows the APU is running, so perhaps U-Boot is hung?:

[pcarr@storm linux]$ xsct
rlwrap: warning: your $TERM is 'xterm-256color' but rlwrap couldn't find it in the terminfo database. Expect some problems.
                                                                                
****** Xilinx Software Commandline Tool (XSCT) v2020.2
  **** SW Build 3064766 on Wed Nov 18 09:12:47 MST 2020
    ** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.


xsct% conn -url 192.168.1.177:3121                                              
tcfchan#0
xsct% ta                                                                        
  1  PS TAP
     2  PMU
        3  MicroBlaze PMU (Sleeping. No clock)
     4  PL
  5  PSU
     6  RPU
        7  Cortex-R5 #0 (Halted)
        8  Cortex-R5 #1 (Lock Step Mode)
     9  APU
       10  Cortex-A53 #0 (Running)
       11  Cortex-A53 #1 (Power On Reset)
       12  Cortex-A53 #2 (Power On Reset)
       13  Cortex-A53 #3 (Power On Reset)
xsct%                                                                           

 

eMMC programming over JTAG (program_flash):

[pcarr@storm linux]$ program_flash -f BOOT.BIN -fsbl zynqmp_fsbl.elf -flash_type emmc -emmc_partition_size large -blank_check -url tcp:192.168.1.177:3121

****** Xilinx Program Flash
****** Program Flash v2020.2 (64-bit)
  **** SW Build (by xbuild) on 2020-11-18-05:13:29
    ** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.


Connected to hw_server @ tcp:192.168.1.177:3121
Available targets and devices:
Target 0 : jsn-JTAG-HS3-210299A8E65D
	Device 0: jsn-JTAG-HS3-210299A8E65D-147e6093-0
	Device 1: jsn-JTAG-HS3-210299A8E65D-5ba00477-0

Retrieving Flash info...

Initialization done, programming the memory
Using default mini u-boot image file - ./399521_zynqmp_emmc.bin.50Mhz
===== mrd->addr=0xFF5E0204, data=0x00000000 =====
BOOT_MODE REG = 0x0000
Downloading FSBL...
Running FSBL...
Finished running FSBL.
===== mwr->addr=0xFF18031C, data=0x64406440 =====
===== mwr->addr=0xFF180314, data=0x01150000 =====
===== mwr->addr=0xFF180318, data=0x00450043 =====
NOTICE:  ATF running on XCZUUNKN/silicon v4/RTL5.1 at 0xfffea000
NOTICE:  BL31: Secure code at 0x60000000
NOTICE:  BL31: Non secure code at 0x10000
NOTICE:  BL31: v1.3(release):
NOTICE:  BL31: Built : 10:38:42, Feb  1 2017
<No further output>

 

Programming eMMC from Vivado Hardware Manager:

Using default mini u-boot image file - ./.Xil/Vivado-366152-storm/zynqmp_emmc.bin.25Mhz
===== mrd->addr=0xFF5E0204, data=0x00000000 =====
BOOT_MODE REG = 0x0000
Downloading FSBL...
Running FSBL...
Finished running FSBL.
===== mwr->addr=0xFF18031C, data=0x64406440 =====
===== mwr->addr=0xFF180314, data=0x01150000 =====
===== mwr->addr=0xFF180318, data=0x00450043 =====
<No further output>

The programming abruptly ends with "[Labtools 27-3161] Flash Programming Unsuccessful"

 

Give kudos if helpful. Accept as solution if it solves your problem.
https://tuxengineering.com/blog

0 Kudos
1 Solution

Accepted Solutions
patocarr
Teacher
Teacher
198 Views
Registered: ‎01-28-2008

Hi,

  The root cause to U-Boot failing to boot was a DDR base address change introduced in 2020.1. In 2019.2, the base address was set to 0x1000000 (CONFIG_SUBSYSTEM_MEMORY_PSU_DDR_0_BANKLESS_BASEADDR) but that changed to 0x0 in 2020.1. Since the project was a port from 2019.2, this change was kept and not updated to 2020.1. There were two other related parameters that changed *BANKLESS_SIZE and *BANKLESS_KERNEL_BASEADDR. Recreating the Petalinux project from scratch highlighted the difference, and boot from JTAG was fixed.

Thanks for all your help @stephenm 

-Pat

 

Give kudos if helpful. Accept as solution if it solves your problem.
https://tuxengineering.com/blog

View solution in original post

13 Replies
watari
Teacher
Teacher
671 Views
Registered: ‎06-16-2013

Hi @patocarr 

 

Are you using secure boot parameter ?

I guess it seems be related to secure mode on CSU.

 

Best regards,

0 Kudos
patocarr
Teacher
Teacher
663 Views
Registered: ‎01-28-2008

Hi @watari 

  Appreciate your prompt response. I am not using secure boot, just standard (non-)security, and no isolation.

 

Thanks,

-Pat

 

Give kudos if helpful. Accept as solution if it solves your problem.
https://tuxengineering.com/blog

0 Kudos
stephenm
Xilinx Employee
Xilinx Employee
531 Views
Registered: ‎09-12-2007

Can you use the --tcl script.tcl switch. For example:

petalinux-boot --jtag --hw_server-url 192.168.1.177:3121 --u-boot --tcl script.tcl 

 

Then use XSCT to boot:

source script.tcl

 

You should have a better idea of what is happening here.

 

0 Kudos
patocarr
Teacher
Teacher
513 Views
Registered: ‎01-28-2008

Thanks for your response, @stephenm 

Dumping the commands into a .tcl file and either executing it or running the commands manually doesn't show anything out of order, but still U-Boot doesn't appear to run. Here's the manual attempt from xsct:

[pcarr@storm DAQ16-2020.2-cfg1]$ xsct
rlwrap: warning: your $TERM is 'xterm-256color' but rlwrap couldn't find it in the terminfo database. Expect some problems.
                                                                                
****** Xilinx Software Commandline Tool (XSCT) v2020.2
  **** SW Build 3064766 on Wed Nov 18 09:12:47 MST 2020
    ** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.


xsct% connect -url 192.168.1.177:3121                                           
tcfchan#0
xsct% targets -set -nocase -filter {name =~ "*PSU*"}                            
xsct% mask_write 0xFFCA0038 0x1C0 0x1C0                                         
xsct% targets -set -nocase -filter {name =~ "*MicroBlaze PMU*"}                 
xsct% dow  "/home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/pmufw.elf"
Downloading Program -- /home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/pmufw.elf
	section, .vectors.reset: 0xffdc0000 - 0xffdc0007
	section, .vectors.sw_exception: 0xffdc0008 - 0xffdc000f
	section, .vectors.interrupt: 0xffdc0010 - 0xffdc0017
	section, .vectors.hw_exception: 0xffdc0020 - 0xffdc0027
	section, .text: 0xffdc0050 - 0xffdd0ad3
	section, .rodata: 0xffdd0ad4 - 0xffdd1df3
	section, .data: 0xffdd1df4 - 0xffdd5e5f
	section, .sdata2: 0xffdd5e60 - 0xffdd5e5f
	section, .sdata: 0xffdd5e60 - 0xffdd5e5f
	section, .sbss: 0xffdd5e60 - 0xffdd5e5f
	section, .bss: 0xffdd5e60 - 0xffdd9b8b
	section, .srdata: 0xffdd9b8c - 0xffdda4a7
	section, .stack: 0xffdda4a8 - 0xffddb4a7
	section, .xpbr_serv_ext_tbl: 0xffddf6e0 - 0xffddfadf
100%    0MB   0.2MB/s  00:00                                                    
Setting PC to Program Start Address 0xffdd0484
Successfully downloaded /home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/pmufw.elf
xsct% Info: MicroBlaze PMU (target 14) Stopped at 0xffd02f24 (Stop)
xsct% con                                                                       
Info: MicroBlaze PMU (target 14) Running (Sleeping. No clock)
xsct% targets -set -nocase -filter {name =~ "*APU*"}                            
xsct% mwr 0xffff0000 0x14000000                                                 
xsct% mask_write 0xFD1A0104 0x501 0x0                                           
xsct% targets -set -nocase -filter {name =~ "*A53*#0"}                          
xsct% source /home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/project-spec/hw-description/psu_init.tcl
xsct% dow  "/home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/zynqmp_fsbl.elf"
Downloading Program -- /home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/zynqmp_fsbl.elf
	section, .text: 0xfffc0000 - 0xfffd061b
	section, .note.gnu.build-id: 0xfffd061c - 0xfffd063f
	section, .init: 0xfffd0640 - 0xfffd0673
	section, .fini: 0xfffd0680 - 0xfffd06b3
	section, .rodata: 0xfffd06c0 - 0xfffd2b1f
	section, .sys_cfg_data: 0xfffd2b40 - 0xfffd3377
	section, .mmu_tbl0: 0xfffd4000 - 0xfffd400f
	section, .mmu_tbl1: 0xfffd5000 - 0xfffd6fff
	section, .mmu_tbl2: 0xfffd7000 - 0xfffdafff
	section, .data: 0xfffdb000 - 0xfffdc1ff
	section, .sbss: 0xfffdc200 - 0xfffdc1ff
	section, .bss: 0xfffdc200 - 0xfffde73f
	section, .heap: 0xfffde740 - 0xfffdeb3f
	section, .stack: 0xfffdeb40 - 0xfffe0b3f
	section, .dup_data: 0xfffe0b40 - 0xfffe1d3f
	section, .handoff_params: 0xfffe9e00 - 0xfffe9e87
	section, .bitstream_buffer: 0xffff0040 - 0xfffffc3f
100%    0MB   0.2MB/s  00:00                                                    
Setting PC to Program Start Address 0xfffc0000
Successfully downloaded /home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/zynqmp_fsbl.elf
xsct% Info: Cortex-A53 #0 (target 10) Stopped at 0xffff0000 (External Debug Request)
xsct% con                                                                       
Info: Cortex-A53 #0 (target 10) Running
xsct% after 4000; stop; catch {stop}; psu_ps_pl_isolation_removal;              
Info: Cortex-A53 #0 (target 10) Stopped at 0xfffce4ec (External Debug Request)  
xsct% psu_ps_pl_reset_config                                                    
xsct% targets -set -nocase -filter {name =~ "*A53*#0"}                          
xsct% dow -data  "/home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/system.dtb" 0x01100000
100%    0MB   0.2MB/s  00:00                                                    
Successfully downloaded /home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/system.dtb
xsct% targets -set -nocase -filter {name =~ "*A53*#0"}                          
xsct% dow  "/home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/u-boot.elf"
Downloading Program -- /home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/u-boot.elf
	section, .data: 0x09000000 - 0x090f7e18
100%    0MB   0.2MB/s  00:04                                                    
Setting PC to Program Start Address 0x09000000
Successfully downloaded /home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/u-boot.elf
xsct% targets -set -nocase -filter {name =~ "*A53*#0"}                          
xsct% dow  "/home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/bl31.elf"
Downloading Program -- /home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/bl31.elf
	section, .text: 0xfffea000 - 0xffff1fff
	section, .rodata: 0xffff2000 - 0xffff2fff
	section, .data: 0xffff3000 - 0xffff67af
	section, stacks: 0xffff67c0 - 0xffff78bf
	section, .bss: 0xffff78c0 - 0xffff865f
	section, xlat_table: 0xffff9000 - 0xffffdfff
	section, coherent_ram: 0xffffe000 - 0xffffefff
100%    0MB   0.2MB/s  00:00                                                    
Setting PC to Program Start Address 0xfffea000
Successfully downloaded /home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/bl31.elf
xsct% con                                                                       
Info: Cortex-A53 #0 (target 10) Running
xsct%                                                                           
xsct% tar                                                                       
  1  PS TAP
     2  PMU
       14  MicroBlaze PMU (Sleeping. No clock)
     3  PL
  5  PSU
     6  RPU
        7  Cortex-R5 #0 (Halted)
        8  Cortex-R5 #1 (Lock Step Mode)
     9  APU
       10* Cortex-A53 #0 (Running)
       11  Cortex-A53 #1 (Power On Reset)
       12  Cortex-A53 #2 (Power On Reset)
       13  Cortex-A53 #3 (Power On Reset)

 

Thanks,

-Pat

 

Give kudos if helpful. Accept as solution if it solves your problem.
https://tuxengineering.com/blog

0 Kudos
stephenm
Xilinx Employee
Xilinx Employee
493 Views
Registered: ‎09-12-2007

NOTICE:  BL31: Secure code at 0x60000000
NOTICE:  BL31: Non secure code at 0x10000
NOTICE:  BL31: v1.3(release):
NOTICE:  BL31: Built : 10:38:42, Feb  1 2017

 

Can you change the dow address for the uboot to 0x10000 in the JTAG TCL script?

0 Kudos
patocarr
Teacher
Teacher
480 Views
Registered: ‎01-28-2008

Hi @stephenm 

  Changing the U-Boot download address didn't change the outcome. This is the changed line:

dow  -data "/home/pcarr/Work/Elk/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/u-boot.elf" 0x10000

 I appreciate your response. Please note the BL31 output you pasted above comes from the program_flash script output. The actual non-secure address where petalinux-boot is uploading U-Boot is never shown.

Thanks,

-Pat

 

Give kudos if helpful. Accept as solution if it solves your problem.
https://tuxengineering.com/blog

0 Kudos
stephenm
Xilinx Employee
Xilinx Employee
476 Views
Registered: ‎09-12-2007

Can you stop the processor and read the PC?

0 Kudos
patocarr
Teacher
Teacher
471 Views
Registered: ‎01-28-2008

Hi @stephenm 

  Stopping the processor (A53#0) shows it's at 0x1c53844. It's not clear what's running in that location.

xsct% tar                                                                       
  1  PS TAP                                                                     
     2  PMU
        3  MicroBlaze PMU (Sleeping. No clock)
     4  PL
  5  PSU
     6  RPU
        7  Cortex-R5 #0 (Halted)
        8  Cortex-R5 #1 (Lock Step Mode)
     9  APU
       10  Cortex-A53 #0 (Running)
       11  Cortex-A53 #1 (Power On Reset)
       12  Cortex-A53 #2 (Power On Reset)
       13  Cortex-A53 #3 (Power On Reset)
xsct% tar 10                                                                    
xsct% stop                                                                      
Info: Cortex-A53 #0 (target 10) Stopped at 0x1c53844 (External Debug Request)   
xsct% state                                                                     
Stopped: (External Debug Request)

Thanks again!

-Pat

 

Give kudos if helpful. Accept as solution if it solves your problem.
https://tuxengineering.com/blog

0 Kudos
stephenm
Xilinx Employee
Xilinx Employee
444 Views
Registered: ‎09-12-2007

You can do an objdump -t u-boot.elf | grep address

 

You can also debug the uboot in Vitis

https://forums.xilinx.com/t5/Design-and-Debug-Techniques-Blog/PetaLinux-Image-Debug-Series-Debugging-ARM-Trusted-Firmware-and/ba-p/1151076

0 Kudos
patocarr
Teacher
Teacher
366 Views
Registered: ‎01-28-2008

Hi@stephenm 

  The ATF seems to be doing its job and passing control, with or without debug information, to U-Boot. The U-Boot version with symbols doesn't run from JTAG boot nor when burned and booted from QSPI (i.e. no output). The normal built U-Boot without symbols boots from QSPI but not from JTAG boot. Perhaps this bit of info may have be clue. Any suggestions how to pursue this further, other than diving into the U-Boot rabbit hole?

 

Thanks,

-Pat

 

 

Give kudos if helpful. Accept as solution if it solves your problem.
https://tuxengineering.com/blog

0 Kudos
stephenm
Xilinx Employee
Xilinx Employee
332 Views
Registered: ‎09-12-2007
xsct% dow -data  "/home/pcarr/Work/DAQ/Petalinux/DAQ16-2020.2-cfg1/images/linux/system.dtb" 0x01100000

Did you change the location of the DTB? in the petalinux project? By default the uboot would expect this at 0x100000 for zynq and zynq ultrascale

However, I would expect an error here.

Anyway, it doesnt look like the uboot is hanged. Can you pass on the uboot elf? the one you used for the testing above

0 Kudos
imok
Visitor
Visitor
308 Views
Registered: ‎03-31-2020

You may have to dive into the rabbit hole...

0 Kudos
patocarr
Teacher
Teacher
199 Views
Registered: ‎01-28-2008

Hi,

  The root cause to U-Boot failing to boot was a DDR base address change introduced in 2020.1. In 2019.2, the base address was set to 0x1000000 (CONFIG_SUBSYSTEM_MEMORY_PSU_DDR_0_BANKLESS_BASEADDR) but that changed to 0x0 in 2020.1. Since the project was a port from 2019.2, this change was kept and not updated to 2020.1. There were two other related parameters that changed *BANKLESS_SIZE and *BANKLESS_KERNEL_BASEADDR. Recreating the Petalinux project from scratch highlighted the difference, and boot from JTAG was fixed.

Thanks for all your help @stephenm 

-Pat

 

Give kudos if helpful. Accept as solution if it solves your problem.
https://tuxengineering.com/blog

View solution in original post