cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Visitor
Visitor
378 Views
Registered: ‎12-16-2019

Debugging RPU While Running PetaLinux

Jump to solution

Hello,

I have a ZCU106 board and I'm trying to run/debug an application on the RPU from Xilinx SDK while simultaneously running PetaLinux (2018.2) on the APU. I've reviewed AR#69143 and I'm interrupting U-Boot in order to update the Linux boot arguments to disable CPU Idle. I'm booting from an SD card and trying to launch the debugger after Linux boots and I've logged in through a UART terminal. When I try launching the 'hello world' sample application from the SDK it's getting stuck on "Executing 'dow <Path To My ELF file>''". At this point my Linux terminal stops responding. After a minute or two of being stuck like this I see the SDK debugger triggering on a breakpoint on "_vector_table" and I'm unable to resume execution.

For my Debug Configuration in the SDK I'm using the Standalone Application Debug profile. The only operations I have selected are "Reset RPU" and "Run psu_init". So I've disabled the steps for programming FPGA, powering up PL, resetting APU, etc. Is there something else that I'm missing? I've been able to successfully run the hello world sample when the board is set to boot from JTAG (with FPGA programming enabled in Debug Configuration) but so far I haven't had success when booting/running Linux from the SD card. Appreciate any help with this.

Jeff

0 Kudos
1 Solution

Accepted Solutions
Highlighted
Moderator
Moderator
218 Views
Registered: ‎05-10-2017

Re: Debugging RPU While Running PetaLinux

Jump to solution

In your lscript.ld (for your rpu application) you can move the regions to tcm. It's a drop-down menu.

If you want to do this by reserving a section of memory, you can carve out a section of ddr as shown below

(example dts below)

/ {
    reserved-memory {
        #address-cells = <2>;
        #size-cells = <2>;
        ranges;
        rproc_0_reserved: rproc@3ed00000 {
            no-map;
            reg = <0x0 0x3ed00000 0x0 0x80000>;
        };
	};
	
	power-domains {
        pd_r5_0: pd_r5_0 {
			#power-domain-cells = <0x0>;
            pd-id = <0x7>;
        };
		
	    pd_tcm_0_a: pd_tcm_0_a {
            #power-domain-cells = <0x0>;
            pd-id = <0xf>;
        };
		
        pd_tcm_0_b: pd_tcm_0_b {
            #power-domain-cells = <0x0>;
            pd-id = <0x10>;
        };
	};
	
    amba {	
        r5_0_tcm_a: tcm@ffe00000 {
            compatible = "mmio-sram";
            reg = <0 0xFFE00000 0x0 0x10000>;
            pd-handle = <&pd_tcm_0_a>;
        };
		
        r5_0_tcm_b: tcm@ffe20000 {
            compatible = "mmio-sram";
            reg = <0 0xFFE20000 0x0 0x10000>;
            pd-handle = <&pd_tcm_0_b>;
        };	
		
		elf_ddr_0: ddr@3ed00000 {
            compatible = "mmio-sram";
            reg = <0 0x3ed00000 0x0 0x80000>;
        };
		
        test_r50: zynqmp_r5_rproc@0 {
            compatible = "xlnx,zynqmp-r5-remoteproc-1.0";
            reg = <0x0 0xff9a0100 0 0x100>, <0x0 0xff9a0000 0 0x100>;
            reg-names = "rpu_base", "rpu_glbl_base";
            dma-ranges;
            core_conf = "split0";
            srams = <&r5_0_tcm_a &r5_0_tcm_b &elf_ddr_0>;
            pd-handle = <&pd_r5_0>;
            interrupt-parent = <&gic>;
            interrupts = <0 29 4>;
  
        };
	};
};

For your rpu application, build it using the same ddr region that you specified in the device-tree i.e. in lscript.ld 

MEMORY
{
   psu_ocm_ram_0_MEM_0 : ORIGIN = 0xFFFC0000, LENGTH = 0x40000
   psu_r5_0_atcm_MEM_0 : ORIGIN = 0x0, LENGTH = 0x10000
   psu_r5_0_btcm_MEM_0 : ORIGIN = 0x20000, LENGTH = 0x10000
   psu_r5_ddr_0_MEM_0 : ORIGIN = 0x3ed00000, LENGTH = 0x80000
   psu_r5_tcm_ram_0_MEM_0 : ORIGIN = 0x0, LENGTH = 0x40000
}
-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------

View solution in original post

6 Replies
Highlighted
Visitor
Visitor
316 Views
Registered: ‎12-16-2019

Re: Debugging RPU While Running PetaLinux

Jump to solution

A bit more information. I've gone back to running just the Hello World program by itself on the RPU and with SW6 set to boot by JTAG. I can get the Hello World program to run in this mode but I'm finding the results to be very inconsistent. In my debug/run configuration I have The following selected:

  • Reset RPU
  • Program FPGA
  • Run psu_init
  • PL Powerup

Starting with the board switched off I first power it up and hit the Run button. The SDK goes through the steps for launching - I see the FPGA being programmed and other steps being executed from the launch scripts. At the very end I see "--- End of Script ---" followed by "Launch script is executed to file .....". But after this nothing happens and I don't see the "Hello World" on my UART terminal. If I click Run again at this point (without power cycling the board) it will sometimes run and send the "Hello World" to my terminal. And sometimes I'm able to do this several times to get the program to run several times in a row each time printing to the terminal. But eventually it fails to run and I get an error message from the SDK. Ive seen a couple of different messages. Usually the error is "AP transaction error, DAP status 30000021" but I have also seen "fpga configuration failed. DONE PIN is not HIGH". At this point I need to power cycle the board and start over. Also, on one occasion the program seemed to run in an infinite loop and continued to repeat the Hello World message on the terminal until reset. I find this hard to explain as I've looked inside the _exit() function and see the infinite loop there where it should stay until I reset and run the program again. Despite seeing the program execute successfully many times I've never seen it run successfully the first time after powering on the board.

I hope this gives a bit more insight into the problem I'm facing. I think maybe I need to figure out why it's behaving so inconsistently even without PetaLinux running on the APU as this may be the underlying cause why I can't get the program to run in parallel to PetaLinux. Appreciate any help.

Jeff

 

0 Kudos
Highlighted
Visitor
Visitor
243 Views
Registered: ‎12-16-2019

Re: Debugging RPU While Running PetaLinux

Jump to solution

Still need some help here.

I've made a couple of changes and I'm having a bit of success now. While running PetaLinux I'm no longer selecting the "Reset RPU" option. I'm also using a Full License for the SDK now instead of an Evaluation license, though I'm not sure that should make a difference. With this setup the Hello World program downloads to the RPU and I see the "Hello World" message on the UART in-between output from PetaLinux. But immediately after the "Hello World" message I'm getting a kernel panic on PetaLinux. So this is what I'm seeing in my UART output:

root@xilinx-zcu106-2018_2:~# while sleep 5; do date; done
Mon Feb 10 19:11:26 UTC 2020
Mon Feb 10 19:11:31 UTC 2020
Mon Feb 10 19:11:36 UTC 2020
Hello World
[ 74.660981] swapper/2[0]: undefined instruction: pc=ffffff80081010c0
[ 74.660984] swapper/1[0]: undefined instruction: pc=ffffff8008101198
[ 74.660990] Code: 00000080 00000000 00000006 00000000 (00000100)
[ 74.660994] Internal error: undefined instruction: 0 [#1] SMP
[ 74.660996] Modules linked in: al5d(O) al5e(O) allegro(O) xlnx_vcu mali(O) uio_pdrv_genirq
[ 74.661012] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O 4.14.0-xilinx-v2018.2 #1
[ 74.661014] Hardware name: ZynqMP ZCU106 RevA (DT)
[ 74.661016] task: ffffffc87b8baf80 task.stack: ffffff80090d8000
[ 74.661024] PC is at tick_irq_enter+0x0/0xe8
[ 74.661028] LR is at irq_enter+0x50/0x70
[ 74.661030] pc : [<ffffff8008101198>] lr : [<ffffff80080a09e8>] pstate: 400001c5
[ 74.661032] sp : ffffff800800bf50
[ 74.661033] x29: ffffff800800bf50 x28: ffffffc87b8baf80
[ 74.661037] x27: 0000000000000000 x26: ffffff800800c000
[ 74.661040] x25: ffffff8008008000 x24: ffffff8008050000
[ 74.661044] x23: ffffff800804f010 x22: ffffff8008deaf78
[ 74.661047] x21: ffffff8008de8000 x20: ffffff8008dd3000
[ 74.661050] x19: ffffff8008dc9000 x18: ffffffc87ff83460
[ 74.661054] x17: 00000000004a5400 x16: ffffff800809b178
[ 74.661057] x15: ffffff8008de8000 x14: 0000000000000000
[ 74.661060] x13: ffffff8008dd3400 x12: 0000000000000001
[ 74.661063] x11: 0000000000000000 x10: 0000000000000880
[ 74.661067] x9 : ffffff80090dbee0 x8 : ffffffc87b8bb860
[ 74.661070] x7 : 0000000000000001 x6 : 00000048771b0000
[ 74.661073] x5 : ffffff80090dbf60 x4 : 0000000000000000
[ 74.661076] x3 : ffffffc87ff7ca04 x2 : 0000000000000000
[ 74.661080] x1 : 0000000000000200 x0 : ffffffc87b8baf80
[ 74.661084] Process swapper/1 (pid: 0, stack limit = 0xffffff80090d8000)
[ 74.661086] Call trace:
[ 74.661089] Exception stack(0xffffff800800be10 to 0xffffff800800bf50)
[ 74.661092] be00: ffffffc87b8baf80 0000000000000200
[ 74.661095] be20: 0000000000000000 ffffffc87ff7ca04 0000000000000000 ffffff80090dbf60
[ 74.661099] be40: 00000048771b0000 0000000000000001 ffffffc87b8bb860 ffffff80090dbee0
[ 74.661102] be60: 0000000000000880 0000000000000000 0000000000000001 ffffff8008dd3400
[ 74.661106] be80: 0000000000000000 ffffff8008de8000 ffffff800809b178 00000000004a5400
[ 74.661109] bea0: ffffffc87ff83460 ffffff8008dc9000 ffffff8008dd3000 ffffff8008de8000
[ 74.661113] bec0: ffffff8008deaf78 ffffff800804f010 ffffff8008050000 ffffff8008008000
[ 74.661117] bee0: ffffff800800c000 0000000000000000 ffffffc87b8baf80 ffffff800800bf50
[ 74.661120] bf00: ffffff80080a09e8 ffffff800800bf50 ffffff8008101198 00000000400001c5
[ 74.661124] bf20: 00000000000001c0 ffffffc87b1bc680 0000008000000000 ffffff80080a0a94
[ 74.661126] bf40: ffffff800800bf50 ffffff8008101198
[ 74.661130] [<ffffff8008101198>] tick_irq_enter+0x0/0xe8
[ 74.661136] [<ffffff80080c1bc4>] scheduler_ipi+0x3c/0x138
[ 74.661140] [<ffffff800808ef14>] handle_IPI+0x10c/0x178
[ 74.661144] [<ffffff8008081554>] gic_handle_irq+0xbc/0xc0
[ 74.661146] Exception stack(0xffffff80090dbe20 to 0xffffff80090dbf60)
[ 74.661150] be20: 0000000000000000 0000000000000000 0000000000000001 0000000000000001
[ 74.661153] be40: 0000000000000000 ffffff80090dbf60 00000048771b0000 0000000000000001
[ 74.661156] be60: ffffffc87b8bb860 ffffff80090dbee0 0000000000000880 0000000000000000
[ 74.661160] be80: 0000000000000001 ffffff8008dd3400 0000000000000000 ffffff8008de8000
[ 74.661163] bea0: ffffff800809b178 00000000004a5400 ffffffc87ff83460 ffffff8008dc9000
[ 74.661167] bec0: ffffff8008de8000 ffffff8008de8000 ffffff8008dd2220 ffffff8008de8b34
[ 74.661170] bee0: 0000000000000000 0000000000000000 ffffffc87b8baf80 0000000000000000
[ 74.661174] bf00: 0000000000000000 ffffff80090dbf60 ffffff8008085384 ffffff80090dbf60
[ 74.661177] bf20: ffffff8008085388 0000000000000145 0000000000000000 0000000000000000
[ 74.661181] bf40: ffffffffffffffff 0000000000000000 ffffff80090dbf60 ffffff8008085388
[ 74.661184] [<ffffff80080830f0>] el1_irq+0xb0/0x140
[ 74.661188] [<ffffff8008085388>] arch_cpu_idle+0x10/0x18
[ 74.661193] [<ffffff80080d0c60>] do_idle+0x120/0x1e0
[ 74.661197] [<ffffff80080d0e90>] cpu_startup_entry+0x20/0x28
[ 74.661201] [<ffffff800808e9d0>] secondary_start_kernel+0x148/0x190
[ 74.661205] Code: 00000080 00000000 00000006 00000000 (00000100)
[ 74.661209] ---[ end trace 06737c0acb7b2239 ]---
[ 74.661212] Kernel panic - not syncing: Fatal exception in interrupt
[ 74.661215] SMP: stopping secondary CPUs
[ 75.063275] Code: 97ffc49c f94526a2 aa1403e1 f9402a60 (b3cb0300)
[ 75.727992] SMP: failed to stop secondary CPUs 1-2
[ 75.732689] Kernel Offset: disabled
[ 75.736161] CPU features: 0x002004
[ 75.739546] Memory Limit: none
[ 75.742587] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
[ 75.749708] swapper/1[0]: undefined instruction: pc=ffffff8008100e18
[ 75.756042] Code: e3031fff e3a0c028 e92d4030 e3a03000 (e3a02020)

I've been assuming Petalinux runs only on the APU - is that correct? I've also been assuming that the memory spaces for the APU and RPU are separate. This is based on the hardware design that comes packaged in the ZCU106 BSP. I haven't made any changes to the hardware design. But obviously something I'm doing is interfering with PetaLinux on the APU. I've also taken the .tcl file that the SDK is using and issued the commands line-by-line through the XSCT console. The kernel panic occurs as soon as the program is downloaded by executing the line:

dow C:/Users/Jeff/xilinx_sdk_workspace/zcu106-test/Debug/zcu106-test.elf

Is downloading the program somehow corrupting memory that PetaLinux is using? I assumed that the APU and RPU have disjoint memory spaces, but maybe that's not true. Any ideas what's going on here?

0 Kudos
Highlighted
Moderator
Moderator
238 Views
Registered: ‎05-10-2017

Re: Debugging RPU While Running PetaLinux

Jump to solution

Is your RPU application running out of ddr or is it in TCM?

If it is in ddr, it may be colliding with the address space in Linux. Can you move it to tcm and then run it?

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Visitor
Visitor
231 Views
Registered: ‎12-16-2019

Re: Debugging RPU While Running PetaLinux

Jump to solution

@jovitac ,

I believe the application is running out of DDR, assuming that's the default configuration.

I've been looking into this further and it's starting to look to me like the APU and RPU are both wired to the DDR and share the same physical address space. So I think when I'm using the debugger to download the program it must be overwriting memory the APU is using resulting in a kernel panic.

Using the TCM might be one option. How would I go about having the RPU use the TCM instead?

I've also been looking into the OpenAMP docs and I think libmetal/OpenAMP may be the route I need to go. I'm working on getting that demo running now. My understanding is that I'm making some edits to the PetaLinux devicetree that will be reserving off some space in the DDR for the RPU to use and also some shared memory between the APU and RPU. Is that correct? Hopefully that will resolve the memory collision I'm seeing.

Finally, I'm trying to figure out how I'll be able to get my code running on the RPU in the end solution without launching it from the debugger. Based on the OpenAMP docs it looks like one option is to use to use the remoteproc framework to start it from the APU. Are there any other options such as booting from flash?

0 Kudos
Highlighted
Moderator
Moderator
219 Views
Registered: ‎05-10-2017

Re: Debugging RPU While Running PetaLinux

Jump to solution

In your lscript.ld (for your rpu application) you can move the regions to tcm. It's a drop-down menu.

If you want to do this by reserving a section of memory, you can carve out a section of ddr as shown below

(example dts below)

/ {
    reserved-memory {
        #address-cells = <2>;
        #size-cells = <2>;
        ranges;
        rproc_0_reserved: rproc@3ed00000 {
            no-map;
            reg = <0x0 0x3ed00000 0x0 0x80000>;
        };
	};
	
	power-domains {
        pd_r5_0: pd_r5_0 {
			#power-domain-cells = <0x0>;
            pd-id = <0x7>;
        };
		
	    pd_tcm_0_a: pd_tcm_0_a {
            #power-domain-cells = <0x0>;
            pd-id = <0xf>;
        };
		
        pd_tcm_0_b: pd_tcm_0_b {
            #power-domain-cells = <0x0>;
            pd-id = <0x10>;
        };
	};
	
    amba {	
        r5_0_tcm_a: tcm@ffe00000 {
            compatible = "mmio-sram";
            reg = <0 0xFFE00000 0x0 0x10000>;
            pd-handle = <&pd_tcm_0_a>;
        };
		
        r5_0_tcm_b: tcm@ffe20000 {
            compatible = "mmio-sram";
            reg = <0 0xFFE20000 0x0 0x10000>;
            pd-handle = <&pd_tcm_0_b>;
        };	
		
		elf_ddr_0: ddr@3ed00000 {
            compatible = "mmio-sram";
            reg = <0 0x3ed00000 0x0 0x80000>;
        };
		
        test_r50: zynqmp_r5_rproc@0 {
            compatible = "xlnx,zynqmp-r5-remoteproc-1.0";
            reg = <0x0 0xff9a0100 0 0x100>, <0x0 0xff9a0000 0 0x100>;
            reg-names = "rpu_base", "rpu_glbl_base";
            dma-ranges;
            core_conf = "split0";
            srams = <&r5_0_tcm_a &r5_0_tcm_b &elf_ddr_0>;
            pd-handle = <&pd_r5_0>;
            interrupt-parent = <&gic>;
            interrupts = <0 29 4>;
  
        };
	};
};

For your rpu application, build it using the same ddr region that you specified in the device-tree i.e. in lscript.ld 

MEMORY
{
   psu_ocm_ram_0_MEM_0 : ORIGIN = 0xFFFC0000, LENGTH = 0x40000
   psu_r5_0_atcm_MEM_0 : ORIGIN = 0x0, LENGTH = 0x10000
   psu_r5_0_btcm_MEM_0 : ORIGIN = 0x20000, LENGTH = 0x10000
   psu_r5_ddr_0_MEM_0 : ORIGIN = 0x3ed00000, LENGTH = 0x80000
   psu_r5_tcm_ram_0_MEM_0 : ORIGIN = 0x0, LENGTH = 0x40000
}
-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------

View solution in original post

Highlighted
Visitor
Visitor
167 Views
Registered: ‎12-16-2019

Re: Debugging RPU While Running PetaLinux

Jump to solution

@jovitac ,

Thank you, that information was very helpful. I made those changes to my device tree and linker scipt and I'm now able to run my program on the RPU without crashing PetaLinux on the APU. I was able to have it execute from both DDR and TCM without any problems.

 

0 Kudos