Linux running on APU with two RPU slaves

[pd92]

Hi all,

I was trying to follow the instructions contained in the following page:

http://www.wiki.xilinx.com/OpenAMP+2017.2

in order to execute Linux on the A53 cores and use the R5 cores as slaves communicating using RPMsg primitives of OpenAMP.

Unfortunately, while the program can be loaded without problems in the R5-0 core, I was not able to use the R5-1 core.

In particular, the command

echo start > /sys/class/remoteproc/remoteproc1/state

has never success.

[  507.651421] remoteproc remoteproc1: powering up ff9a0200.zynqmp_r5_rproc
[  507.658590] remoteproc remoteproc1: Booting fw image matrix_mult_r5_1.elf, size 630016
[  507.668209] remoteproc remoteproc1: failed to declare rproc mem as DMA mem.
[  507.675099] remoteproc remoteproc1: Failed to declare rproc memory resource: -12
[  507.682547] remoteproc remoteproc1: Boot failed: -12

I presume the problem lies in the relation between the linker script lscript.ld, the rsc_table.c file and the system-user.dtsi files. In particular, I am not 100% sure about the meaning of the following two points:

• Check that the application linker script (lscript.ld) addresses match and fit the DTS zynqmp_r5_rproc memory sections.
• Check that the appplication rsc_table.c address for RSC_RPROC_MEM carveout is not overlapping the linker script addresses.

The application is a simple matrix multiplication, that works fine when only R5-0 is involved.

Is it available anywhere an example of code and configuration that would really work in this configuration?

BR

[jesuslazaro]

Hi,

I am trying to do the same but with the echo example. It looks like both have the same problems.

I have tried to make all the changes, although I am a bit confused with the linker script part. Looking into the 2016.1 demo, it looks like more things should be changed.

For example, my example tries to set up virtio0 instead of virtio1:

[   19.539819] remoteproc remoteproc1: powering up ff9a0200.zynqmp_r5_rproc
[   19.547548] remoteproc remoteproc1: Booting fw image OpenAMP, size 784592
[   19.557192] remoteproc remoteproc1: registered virtio0 (type 7)
[   19.558850] virtio_rpmsg_bus virtio0: rpmsg host is online
[   19.559181] zynqmp_gpd_attach_dev error -13, node 36
[   19.580349] remoteproc remoteproc1: bad phdr da 0x3ed00000 mem 0x31c00
[   19.586825] remoteproc remoteproc1: Failed to load program segments: -22
[   19.593729] remoteproc remoteproc1: Boot failed: -22

Does anybody have any hints or examples?

[jovitac]

@pd92 

• Check that the application linker script (lscript.ld) addresses match and fit the DTS zynqmp_r5_rproc memory sections.

Here, the base address of psu_ddr_S_AXI_BASEADDR in the lscript.ld should be changed to match the memory address of elf_ddr_0 for R5-0 firmware and elf_ddr_1 for R5-1 firmware in the device-tree.

i.e. psu_ddr_S_AXI_BASEADDR will have a base address of 0x3ED00000 for R5-0 and 0x3ED40000 for R5-1

• Check that the application rsc_table.c address for RSC_RPROC_MEM carveout is not overlapping the linker script addresses

Here you will need to change the following 

In rsc_table.c check that RING_TX, RING_RX and TSC_RPROC_MEM entries are within reserved memory section in the device tree but do not overlap with sections (DDR for RPU0/1, vring device nodes etc). This is mentioned in the OpenAMP 2017.3 wiki

Hope this helps.

[ultra-hsandlar]

Hi,

I've been working on following these directions as well and have made it further than you, but have still not been able to get R5-1 functioning as expected.

In the example openamp split-mode dtsi, I adjusted the reserved DDR memory space for each RPU such that the TX/RX_RING buffers will not overlap between R5-0 and R5-1 when running the application. These changes look as follows:

elf_ddr_0: ddr@3ed00000 {
compatible = "mmio-sram";
reg = <0 0x3ed00000 0x0 0x80000>;
};

elf_ddr_1: ddr@3ed80000 {
compatible = "mmio-sram";
reg = <0 0x3ed80000 0x0 0x80000>;
};

I then went into the SDK project and customized R5-1's rsc_table file to reflect this device-tree change --

#define RING_TX 0x3ED80000
#define RING_RX 0x3ED84000

{RSC_RPROC_MEM, 0x3ed80000, 0x3ed80000, 0x100000, 0},

I also went into the lscript.ld for R5-1 and updated the "psu_ddr_S_AXI_BASEADDR" to 0x3ED80000.

After making these changes, I'm able to start each RPU but I only get a /dev entry for rpmsg0. I have not yet been able to figure out why the rpmsg channel creation is failing for R5-1. I also noticed that if I attempt to load R5-1 before R5-0, the entire system hangs. Without the rpmsg dev entry for R5-1, I'm unable to run the echo_test against R5-1.

root@zcu102-zynqmp:~# echo image_echo_test_r5_0 > /sys/class/remoteproc/remoteproc0/firmware
root@zcu102-zynqmp:~# echo image_echo_test_r5_1 > /sys/class/remoteproc/remoteproc1/firmware
root@zcu102-zynqmp:~# echo start > /sys/class/remoteproc/remoteproc0/state
[ 23.100603] remoteproc remoteproc0: powering up ff9a0100.zynqmp_r5_rproc
[ 23.172086] remoteproc remoteproc0: Booting fw image image_echo_test_r5_0, size 980788
[ 23.183012] remoteproc remoteproc0: registered virtio0 (type 7)
[ 23.189012] zynqmp_r5_remoteproc ff9a0100.zynqmp_r5_rproc: RPU boot from TCM.
[ 2Starting application...
Try to init remoteproc resource
3.196611] remoteproc remoteproc0: remote processor ff9a0100.zynqmp_r5_rproc is now up
root@zcu102-zynqmp:~# [ Init remoteproc resource succeeded
Waiting for events...
23.212623] virtio_rpmsg_bus virtio0: rpmsg host is online
[ 23.212871] virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x1

root@zcu102-zynqmp:~# echo start > /sys/class/remoteproc/remoteproc1/state
[ 27.764598] remoteproc remoteproc1: powering up ff9a0200.zynqmp_r5_rproc
[ 27.821343] remoteproc remoteproc1: Booting fw image image_echo_test_r5_1, size 774836
[ 27.831421] virtio_rpmsg_bus virtio1: rpmsg host is online
[ 27.836853] remoteproc remoteproc1: registered virtio1 (type 7)
[ 27.842808] zynqmp_r5_remoteproc ff9a0200.zynqmp_r5_rproc: RPU boot from TCM.
[ Starting application...
Try to init remoteproc resource
27.850394] remoteproc remoteproc1: remote processor ff9a0200.zynqmp_r5_rproc is now up
root@zcu102-zynqmp:~#
root@zcu102-zynqmp:~# modprobe rpmsg_user_dev_driver
[ 36.793037] rpmsg_user_dev_driver: loading out-of-tree module taints kernel.
[ 36.800443] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: rpmsg_user_dev_rpmsg_drv_probe
[ 36.809689] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: new channel: 0x400 -> 0x1!
root@zcu102-zynqmp:~# ls /dev/rpmsg*
/dev/rpmsg0

Any input would be appreciated.

Thanks,

Holden

[jovitac]

Load rpmsg_user_dev_driver LKM before starting RPU-0 and RPU-1 - step-4 in the wiki

http://www.wiki.xilinx.com/OpenAMP+2017.3

[ultra-hsandlar]

@jovitac -- Same result. Depending on the doc you look at some show loading the rpmsg_user_dev_driver prior to starting the RPU's, other docs show to load the driver after. Here's the log:

root@zcu102-zynqmp:~# cat test_openamp.sh
set -x
modprobe rpmsg_user_dev_driver
echo image_echo_test_r5_0 > /sys/class/remoteproc/remoteproc0/firmware
echo image_echo_test_r5_1 > /sys/class/remoteproc/remoteproc1/firmware
echo start > /sys/class/remoteproc/remoteproc0/state
sleep 2
echo start > /sys/class/remoteproc/remoteproc1/state
root@zcu102-zynqmp:~# ./test_openamp.sh
++ modprobe rpmsg_user_dev_driver
[ 11.947983] rpmsg_user_dev_driver: loading out-of-tree module taints kernel.
++ echo image_echo_test_r5_0
++ echo image_echo_test_r5_1
++ echo s[ 11.956264] remoteproc remoteproc0: powering up ff9a0100.zynqmp_r5_rproc
tart
[ 12.043856] remoteproc remoteproc0: Booting fw image image_echo_test_r5_0, size 980788
[ 12.054778] remoteproc remoteproc0: registered virtio0 (type 7)
[ 12.057670] virtio_rpmsg_bus virtio0: rpmsg host is online
[ 12.066201] zynqmp_r5_remoteproc ff9a0100.zynqmp_r5_rproc: RPU boot from TCM.
[ 1Starting application...
Try to init remoteproc resource
Init remoteproc resource succeeded
Waiting for events...
2.073791] remoteproc remoteproc0: remote processor ff9a0100.zynqmp_r5_rproc is now up
[ 12.078431] virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x1
[ 12.078523] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: rpmsg_user_dev_rpmsg_drv_probe
[ 12.078680] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: new channel: 0x400 -> 0x1!
++ sleep 2
++ echo start
[ 14.120149] remoteproc remoteproc1: powering up ff9a0200.zynqmp_r5_rproc
[ 14.185839] remoteproc remoteproc1: Booting fw image image_echo_test_r5_1, size 779684
[ 14.195905] virtio_rpmsg_bus virtio1: rpmsg host is online
[ 14.201342] remoteproc remoteproc1: registered virtio1 (type 7)
[ 14.207283] zynqmp_r5_remoteproc ff9a0200.zynqmp_r5_rproc: RPU boot from TCM.
[ Starting application...
After init_system();
Try to init remoteproc resource
14.214870] remoteproc remoteproc1: remote processor ff9a0200.zynqmp_r5_rproc is now up
root@zcu102-zynqmp:~# ls /dev/rp*
/dev/rpmsg0

[jovitac]

Hi Holden,

For R5-1,if your "psu_ddr_S_AXI_BASEADDR" is 0x3ED80000 the RING_TX value should be different

You can make to below changes to R5-1

#define RING_TX 0x3EE80000
#define RING_RX 0x3EE84000

{RSC_RPROC_MEM, 0x3ee80000, 0x3ee80000, 0x100000, 0},

[ultra-hsandlar]

@jovitac -- Thanks. That was definitely part of the problem. I now get two /dev/rpmsg entries after starting each RPU and loading the kernel module.

However, if I then run the echo_test concurrently on the processors R5-1 stalls. R5-0 executes without any issues -- I can re-run as many times as I want as well. On R5-1 I see the following:

root@zcu102-zynqmp:~# echo_test -d /dev/rpmsg1

Echo test start

Open rpmsg dev!

****************************************
Please enter command and press enter key
****************************************
1 - Send data to remote core, retrieve the echo and validate its integrity ..
2 - Quit this application ..
CMD>1

sending payload number 0 of size 17
echo test: sent : 17

FYI I am connected via console to observe R5-0, and ssh to observe R5-1. Any advice?

Thanks,

Holden

[jovitac]

It shouldn't stall. Could you make sure that both demo channels are created? You can stop RPU-0 and RPU-1, reload the driver and then start RPU-0 and RPU-1. 

[ultra-hsandlar]

@jovitac --

I see both channels created per these messages:

[ 15.384034] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: rpmsg_user_dev_rpmsg_drv_probe
[ 15.384210] rpmsg_user_dev_driver virtio0:rpmsg-openamp-demo-channel: new channel: 0x400 -> 0x1!

[ 17.526866] rpmsg_user_dev_driver virtio1:rpmsg-openamp-demo-channel: rpmsg_user_dev_rpmsg_drv_probe
[ 17.527018] rpmsg_user_dev_driver virtio1:rpmsg-openamp-demo-channel: new channel: 0x400 -> 0x1!

I also see two rpmsg devices --

root@zcu102-zynqmp:~# ls /dev/rp*
/dev/rpmsg0 /dev/rpmsg1

One curious observation is that when telling the R5-1 to start executing the echo test (against /dev/rpmsg1) I see some garbage data come out of R5-0. I'm guessing this is going to be an overlapped address space issue, though I'm not really sure where.

After the hang, I can CTRL+C out of the echo_test application, then do the following:

root@zcu102-zynqmp:~# modprobe -r rpmsg_user_dev_driver
[ 183.374391] rpmsg_user_dev_driver virtio1:rpmsg-openamp-demo-channel: Removing rpmsg user dev.
[ 183.383067] rpmsg_user_dev_driver rpmsg1: Releasing rpmsg user dev device.
root@zcu102-zynqmp:~# echo stop > /sys/class/remoteproc/remoteproc0/state
[ 198.068866] remoteproc remoteproc0: stopped remote processor ff9a0100.zynqmp_r5_rproc
root@zcu102-zynqmp:~# echo stop > /sys/class/remoteproc/remoteproc1/state
[ 200.004829] remoteproc remoteproc1: stopped remote processor ff9a0200.zynqmp_r5_rproc
root@zcu102-zynqmp:~# echo start > /sys/class/remoteproc/remoteproc0/state
[ 209.772563] remoteproc remoteproc0: powering up ff9a0100.zynqmp_r5_rproc
[ 209.780075] remoteproc remoteproc0: Booting fw image image_echo_test_r5_0, size 980788
[ 209.791268] virtio_rpmsg_bus virtio0: rpmsg host is online
[ 209.796702] remoteproc remoteproc0: registered virtio0 (type 7)
[ 209.802689] zynqmp_r5_remoteproc ff9a0100.zynqmp_r5_rproc: RPU boot from TCM.
[ 20Starting application...
Try to init remoteproc resource
Init remoteproc resource succeeded
Waiting for events...
9.810278] remoteproc remoteproc0: remote processor ff9a0100.zynqmp_r5_rproc is now up
[ 209.828188] virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x1
root@zcu102-zynqmp:~# echo start > /sys/class/remoteproc/remoteproc1/state
[ 214.612572] remoteproc remoteproc1: powering up ff9a0200.zynqmp_r5_rproc
[ 214.619905] remoteproc remoteproc1: Booting fw image image_echo_test_r5_1, size 779684
[ 214.629698] remoteproc remoteproc1: rvdev idr_alloc failed: -28
[ 214.635552] remoteproc remoteproc1: Failed to handle vdev resources: -28
[ 214.642320] remoteproc remoteproc1: Boot failed: -28
-sh: echo: write error: No space left on device

As you can see, R5-1 fails to come back up. I'm definitely grasping at straws on how to debug this issue -- In general I've spent the past several days trying to get this stock example working.

Thanks,

Holden

[jovitac]

Does seem to be an issue with overlap address space.

Attached is the device-tree I have, and rsc_table.c and lscript.ld for both R5-0 and R5-1. Hope this helps

[ultra-hsandlar]

Hi @jovitac

I really appreciate the quick replies. Thanks for the files. I have a couple of observations -- 

I copied in your files to each project directory and went to rebuild. R5-1's project built find, but R5-0's project had psu_ddr_S_AXI_BASEADDR size set to 0x00020000, which the SDK complained was not enough space to fit my .bss region: 

Description Resource Path Location Type
echo_test_r5_0.elf section `.bss' will not fit in region `psu_ddr_S_AXI_BASEADDR' echo_test_r5_0 C/C++ Problem

I adjusted that size value to 0x00040000, rebuilt and placed the new binaries on the device. 

I've matched my device tree include to yours.

I am still getting hanging issues. It seems that I can start echo_test on both RPU's, but whichever one I kick-off executing the application code first will continue to work. The other will hang. If I start one test executing and wait for it to complete and report 0 errors, then I can get both R5's to pass the test -- however, this is not concurrent and I still think indicates a memory space overlap somewhere.

I can unload the kernel module, stop the R5's then start them back up and execute the echo_test again -- but I see the same problem consistently... whichever RPU starts executing the test first will continue to work as expected, but the other will stall.

Thanks,

Holden

[jovitac]

Which version of the tools are you on?

For RPU-1

• Edit platform_info.h and replace IPI_IRQ_VECT_ID value 65 by 66
• Edit platform_info.c and replace IPI_BASE_ADDR value 0xFF310000 by 0xFF320000

[ultra-hsandlar]

I'm on 2017.3 tools. 

In this version of the tools, platform_info.h contains 

#define IPI_IRQ_VECT_ID         XPAR_XIPIPSU_0_INT_ID

Which I can right click on and "Open Declaration" --

xparameters.h contains:

#define  XPAR_XIPIPSU_0_INT_ID XPAR_PSU_IPI_2_INT_ID

#define  XPAR_PSU_IPI_2_INT_ID  66U

So, IPI_IRQ_VECT_ID looks correct. Here's the same process for IPI_BASE_ADDR:

#define IPI_BASE_ADDR        XPAR_XIPIPSU_0_BASE_ADDRESS /* IPI base address*/

#define  XPAR_XIPIPSU_0_BASE_ADDRESS XPAR_PSU_IPI_2_BASE_ADDRESS

#define  XPAR_PSU_IPI_2_BASE_ADDRESS  0xFF320000U

All of this looks to be as expected. Any other suggestions?

Thanks again,

Holden

[jovitac]

Can you copy both your console logs, I believe it is just waiting for user command input and it's not stuck. I just tried this out on 2017.3

[ultra-hsandlar]

Hi jovitac,

I'm attaching the two console logs. Note that the R5-0 was captured via UART and R5-1 was captured via ssh. Let me know if you see something I'm missing.

Thanks,

Holden

[kevin.kruse]

I've been working from Holden's sources and was able to get the r5 echo test running simultaneously on both r5_0 and r5_1, so I thought I'd share what needed fixing. To limit the number of "gotcha"s, I recreated the two SDK projects (and their BSPs) from scratch, following http://www.wiki.xilinx.com/OpenAMP+2017.3 . What changed:

1. I was amused to find that Xilinx's meta-openamp repo, specifically echo_test.c for the APU, was updated recently (4 hours ago at the time of writing). So someone has been looking into this stuff and realized it needed fixing. Previously, error messages pointed to the wrong /dev/ entry and the echo_test options -d and -n didn't quite work. Also, the user prompt shown in the wiki was removed, so I guess that wiki image needs updating.

2. The wiki is vague in describing exactly the address changes needed to support both r5 processors, so I'll be explicit with how my addresses are configured. Most notably, I had to change the r5_0 rsc_table.c addresses to not overlap the elf_ddr_1 space in the device tree. From my reading of the wiki, I only understood that I needed to change the r5_1 rsc_table.c addresses. But if you follow the wiki's device tree, you need to change both rsc_table.c files.

Aside: I don't know why the RSC_RPROC_MEM space is so large compared to the size of the RING_RX and RING_TX within it, but let's just play along.

So in conclusion, Xilinx has cleaned up some issues in the APU application echo_test.c, and I think their next task should be to make the wiki more specific. It doesn't hurt to be explicit if it'll help someone down the road. Also, the images in the wiki need to be updated to reflect the removal of the user prompt in echo_test.

I've included my relevant addresses below, and attached the console logs for r5_0 (captured over uart) and r5_1 (captured over ssh). As proof that these worked concurrently, you can find in the r5_0 log that during the echo_test, the second echo_test was started:

 sending payload numb[  332.922896] rpmsg_user_dev_driver virtio1:rpmsg-openamp-demo-channel: Sent init_.
er 36 of size 52                                                                                         
echo test: sent : 52                                                                                     
 received payload number 36 of size 52 

ADDRESSES:

rsc_table.c (r5_0):

#define RING_TX                     0x3ED80000
#define RING_RX                     0x3ED84000

[...]

{RSC_RPROC_MEM, 0x3ed80000, 0x3ed80000, 0x100000, 0},

 lscript.ld (r5_0):

 psu_ddr_S_AXI_BASEADDR

Base Address = 0x3ED00000

Size = 0x00040000

rsc_table.c (r5_1):

#define RING_TX                     0x3EE80000
#define RING_RX                     0x3EE84000

[...]

{RSC_RPROC_MEM, 0x3ee80000, 0x3ee80000, 0x100000, 0},

 lscript.ld (r5_1):

psu_ddr_S_AXI_BASEADDR

Base Address = 0x3ed40000

Size = 0x00040000

device tree:

    reserved-memory {
        #address-cells = <2>;
        #size-cells = <2>;
        ranges;
        rproc_0_reserved: rproc@3ed00000 {
            no-map;
            reg = <0x0 0x3ed00000 0x0 0x1000000>;
        };
    };

[...]

        elf_ddr_0: ddr@3ed00000 {
            compatible = "mmio-sram";
            reg = <0 0x3ed00000 0x0 0x40000>;
        };

        elf_ddr_1: ddr@3ed40000 {
            compatible = "mmio-sram";
            reg = <0 0x3ed40000 0x0 0x40000>;
        };

[jovitac]

Thanks for the feedback kevin.kruse

The changes that you are referring to are present in master branch. These are for the 2018.1 release. The image for that will be released with the 2018.1 wiki

In case of the address ranges for the rsc_table, we will update the wiki so that it is more clear to the user what changes need to be made.

Your response and feedback are much appreciated.

Best regards

[leopoldo]

Hi jovitac ,

I'm using

Vivado SDK 2017.4

Petalinux 2017.4

I'm writing a custom application modified , adding code , in FreeRTOS FW of matrix_multiply demo.

OpenAMP UG1186

matrix_multiply_demo defines

ps7_ddr0_S_AXI_BASEADDR in lscript.ld as :

Base addr 0x3e000000  Size 0x00400000

Adding code to demo, there is error of overflow ddr0

How do I modify system-user.dtsi , openamp-overlay.dtsi and lscript.ld ?

Actually :

system-user.dtsi

/include/ "system-conf.dtsi"
/ {
    reserved-memory {
        #address-cells = <1>;
        #size-cells = <1>;
        ranges;
        rproc_0_reserved: rproc@3e000000 {
            no-map;
            reg =  <0x3e000000 0x01000000>;
        };
    };

    amba {
        elf_ddr_0: ddr@ {
            compatible = "mmio-sram";
            reg = <0x3e000000 0x400000>;
        };
    };
    remoteproc0: remoteproc@0 {
        compatible = "xlnx,zynq_remoteproc";
        firmware = "FreeRTOS";
        vring0 = <15>;
        vring1 = <14>;
        srams = <&elf_ddr_0>;

    };
};

and openamp-overlay.dtsi

/ {
    reserved-memory {
        #address-cells = <1>;
        #size-cells = <1>;
        ranges;
        rproc_0_reserved: rproc@3e000000 {
            no-map;
            reg =  <0x3e000000 0x01000000>;
        };
    };

    amba {
        elf_ddr_0: ddr@0 {
            compatible = "mmio-sram";
            reg = <0x3e000000 0x400000>;
        };
    };
    remoteproc0: remoteproc@0 {
        compatible = "xlnx,zynq_remoteproc";
        firmware = "FreeRTOS";
        vring0 = <15>;
        vring1 = <14>;
        srams = <&elf_ddr_0>;

    };
};