UG1186 v2019.2 OpenAMP Demo fails

[dave74321]

Using 2019.2 tools to follow the OpenAMP Echo Test demo in UG1186 v2019.2.

Very basic Vivado design created on Windows with Petalinux project created in Ubuntu.

Attempting to run image_echo_test and viewing linux output on serial port uart 1, however it fails to work

Any ideas on what I should do to get this to work?

[dave74321]

I still don't know what I'm doing wrong...I'm just trying to get the example in UG1186 (2019.2) to work.

I can see in the boot log that 

[ 0.000000] OF: reserved mem: initialized node rproc@0x3ed40000, compatible id shared-dma-pool
[ 0.000000] cma: Failed to reserve 1000 MiB

But I don't know what possible reasons could be causing this, so if anyone has any ideas??

Thanks

[jovitac]

Can you do a /proc/meminfo in linux and check  "CMA Total" and "CMA free"

[dave74321]

Hi @jovitac, thanks for your reply.

After login to the booted Linux on my custom board I typed: "cat /proc/meminfo", which gave this result:

MemTotal:        2044088 kB
MemFree:         1970648 kB
MemAvailable:    1940164 kB
Buffers:              44 kB
Cached:            30620 kB
SwapCached:            0 kB
Active:            16560 kB
Inactive:          20612 kB
Active(anon):      16520 kB
Inactive(anon):    20608 kB
Active(file):         40 kB
Inactive(file):        4 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          6556 kB
Mapped:             5216 kB
Shmem:             30620 kB
Slab:              24704 kB
SReclaimable:       6568 kB
SUnreclaim:        18136 kB
KernelStack:        1344 kB
PageTables:          344 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1022044 kB
Committed_AS:      66808 kB
VmallocTotal:   263061440 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:              720 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB

Thanks.

[jovitac]

Your CMA memory is 0k which means no memory got allocated. You device-tree nodes are incorrect. When you specify rproc address for dma and reserved memory just give it as rproc@3ed40000 and rproc@3ed00000

	reserved-memory {
		#address-cells = <2>;
		#size-cells = <2>;
		ranges;
		rproc_0_dma: rproc@3ed40000 {
			no-map;
			compatible = "shared-dma-pool";
			reg = <0x0 0x3ed40000 0x0 0x100000>;
		};
		rproc_0_reserved: rproc@3ed00000 {
			no-map;
			reg = <0x0 0x3ed00000 0x0 0x40000>;
		};
	};

[dave74321]

Hi @jovitac, thanks for your reply.

I just tried it again with the device tree you suggest rather than the one from UG1186, but "cat /proc/meminfo" still indcates the same with CMA as 0k...Anything else that could be causing this?

Below shows part of the device tree as listed after stopping Uboot - with the different node names for rproc reserved memory.

Thanks.

[jovitac]

Is this on an eval board or is it custom board? How much ddr memory is available on the board?

Does this work when openamp related nodes are removed from the dts?

[dave74321]

Hi @jovitac,

Custom board with 0x80000000 DDR RAM

I'll check on CMA memory tomorrow when at work.

Thanks.

[dave74321]

Hi @jovitac,

I removed the device tree code that I had copied from UG1186 into system-user.dtsi.  So now system-user.dtsi now only contains:

/include/ "system-conf.dtsi"
/ {
};

I did a "petalinux-build -x mrproper -f", "petalinux-build" and "petalinux-package..."

Then booted my cusom board, logged in and typed "cat /proc/meminfo" this returned

MemTotal:        2045404 kB
MemFree:         1904208 kB
MemAvailable:    1873700 kB
Buffers:              44 kB
Cached:            30608 kB
SwapCached:            0 kB
Active:            16728 kB
Inactive:          20740 kB
Active(anon):      16688 kB
Inactive(anon):    20736 kB
Active(file):         40 kB
Inactive(file):        4 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          6828 kB
Mapped:             5236 kB
Shmem:             30608 kB
Slab:              24856 kB
SReclaimable:       6516 kB
SUnreclaim:        18340 kB
KernelStack:        1440 kB
PageTables:          344 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1022700 kB
Committed_AS:      68532 kB
VmallocTotal:   263061440 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:              720 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:        1024000 kB
CmaFree:          956936 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB

Thanks for your help on this...

[dave74321]

Hi @jovitac, 

I would be grateful for any ideas?...

Thanks!

[jovitac]

Hi,

Could you ty changing the CMA

petalinux-config -c kernel

Go to Device Drivers / Generic Driver options
Make sure the following settings are met:
o DMA Contiguous Memory Allocator
 True
o Size in Mega Bytes

(I have mine set to 256). From your boot logs, it seems like it is trying to reserve 1000MB

[dave74321]

Hi @jovitac, Thanks for your reply.

Manually setting the Contiguous Memory Allocator size to 256MB.  And going back to having OpenAmp device tree settings in system-user.dtsi

Then booted my customboard, logged in and typed "cat /proc/meminfo" returned this:

MemTotal:        2044088 kB
MemFree:         1907556 kB
MemAvailable:    1876908 kB
Buffers:              44 kB
Cached:            30608 kB
SwapCached:            0 kB
Active:            16844 kB
Inactive:          20580 kB
Active(anon):      16804 kB
Inactive(anon):    20576 kB
Active(file):         40 kB
Inactive(file):        4 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          6752 kB
Mapped:             5164 kB
Shmem:             30608 kB
Slab:              23552 kB
SReclaimable:       6240 kB
SUnreclaim:        17312 kB
KernelStack:        1260 kB
PageTables:          332 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1022044 kB
Committed_AS:      67888 kB
VmallocTotal:   263061440 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:              720 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:         262144 kB
CmaFree:          195080 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB

So that seems a bit better - thank you!

However, previously the boot log listed "SMP: Total of 4 processors activated.", now though there is "SMP: Total of 1 processors activated."  I guess this is referring to the A53 cores?  And the OpenAMP example is targeting R5 anyway, and remoteproc starts say "powering up r5@0", so I guess this may not be relevant?

But the RemoteProc still fails to work...

As I am not using the Xilinx BSP, as the product I'm developing does not use a Xilinx board, are there perhaps more settings, that are not listed in UG1186, that have to be set in the PetaLinux config for OpenAMP to work??

Thanks again for any help you might be able to give on this!!

[jovitac]

Can you do an lsmod? You should had rpmsg_char driver enabled.

Check if you have the following drivers enabled in the kernel.

Symbol: UIO [=y]                                                                                                                                                                      │  
  │ Type  : tristate  
Prompt: Userspace I/O drivers │ 
│ Location: │ 
│ (1) -> Device Drivers

Symbol: VIRTIO [=m] │ 
│ Type : tristate │ 
│ Defined at drivers/virtio/Kconfig:1 │ 
│ Selected by [m]: │ 
│ - REMOTEPROC [=m] && HAS_DMA [=y] │ 
│ - RPMSG_VIRTIO [=m] && HAS_DMA [=y]

│ Symbol: RPMSG [=m] │ 
│ Type : tristate │ 
│ Defined at drivers/rpmsg/Kconfig:6 │ 
│ Selected by [m]: │ 
│ - RPMSG_VIRTIO [=m] && HAS_DMA [=y] │ 

│ Symbol: RPMSG_CHAR [=m] │ 
│ Type : tristate │ 
│ Prompt: RPMSG device interface │ 
│ Location: │ 
│ -> Device Drivers │ 
│ (1) -> Rpmsg drivers │ 
│ Defined at drivers/rpmsg/Kconfig:9 │ 
│ Depends on: RPMSG [=m] && NET [=y] │ 
│

[dave74321]

Hi @jovitac, Thank you for your reply.  

I logged in and typed lsmod on my custom board and there seemed to be no mention of rpmsg_char.

root@test_design:~# lsmod
    Tainted: G
dmaproxy 16384 0 - Live 0xffffff8000b86000 (O)
al5e 20480 0 - Live 0xffffff8000b7c000 (O)
al5d 16384 0 - Live 0xffffff8000b61000 (O)
allegro 40960 2 al5e,al5d, Live 0xffffff8000b4f000 (O)
xlnx_vcu 16384 1 allegro, Live 0xffffff8000b47000
xlnx_vcu_clk 20480 0 - Live 0xffffff8000b3e000
zynqmp_r5_remoteproc 20480 0 - Live 0xffffff8000b34000
remoteproc 40960 1 zynqmp_r5_remoteproc, Live 0xffffff8000b23000
virtio_ring 20480 1 remoteproc, Live 0xffffff8000b19000
virtio 16384 1 remoteproc, Live 0xffffff8000b10000
xlnx_vcu_core 16384 4 - Live 0xffffff8000b08000
uio_pdrv_genirq 16384 0 - Live 0xffffff8000b00000
root@test_design:~#

How do I get the PetaLinux tool to include it?

I already had setting as shown below

[jovitac]

petalinux-config -c kernel

│ Symbol: RPMSG_CHAR [=m] │ 
│ Type : tristate │ 
│ Prompt: RPMSG device interface │ 
│ Location: │ 
│ -> Device Drivers │ 
│ (1) -> Rpmsg drivers │ 
│ Defined at drivers/rpmsg/Kconfig:9 │ 
│ Depends on: RPMSG [=m] && NET [=y] │ 
│

[dave74321]

Hi @jovitac ,

Please could you confirm if you are saying there is a RPMSG_CHAR setting to select under the Petalinux kernel config menu?

As per my last post JPEG screen shot of my Petalinux menu, there is no RPMSG_CHAR setting to select under menu Device Drivers->RPMSG Drivers.

There is only RPMSG device interface, which was already selected as "M", and Virtio RPMSG bus driver which also had "M".  There is also Qualcom RPM Glink driver but I had not selected that.

Is there any chance you could show me a screen shot of where RPMSG_CHAR is in the Petalinux  config menu?

Thank you!

[jovitac]

Hi. I think you should probably have it enabled.

Do a help on the rpmsg device interface, you should be able to see it.

RPMSG device interface
│ CONFIG_RPMSG_CHAR: │
│ │
│ Say Y here to export rpmsg endpoints as device files, usually found │
│ in /dev. They make it possible for user-space programs to send and │
│ receive rpmsg packets.
│ │
│ Symbol: RPMSG_CHAR [=m] │
│ Type : tristate │
│ Prompt: RPMSG device interface │
│ Location: │
│ -> Device Drivers │
│ -> Rpmsg drivers │
│ Defined at drivers/rpmsg/Kconfig:9 │
│ Depends on: RPMSG [=m] && NET [=y]

If the module does not get loaded when linux boots, do a modprobe rpmsg_char and then run the demo.

[dave74321]

Hi @jovitac,

How do I "do a help on the rpmsg device interface"?

I booted up and logged into my custom board and then entered the command "modprobe rpmsg_char" but it still freezes up - please see below:

PetaLinux 2019.2 test_design /dev/ttyPS0

test_design login: root
Password:
root@test_design:~# modprobe rpmsg_char
root@test_design:~# modprobe zynqmp_r5_remoteproc
root@test_design:~# echo image_echo_test > /sys/class/remoteproc/remoteproc0/firmware
root@test_design:~# echo start > /sys/class/remoteproc/remoteproc0/state
[   44.704938] remoteproc remoteproc0: powering up r5@0
[   44.711048] remoteproc remoteproc0: Booting fw image image_echo_test, size 839052

Any other thoughts why the UG1186 v2019.2 OpenAMP Demo does not work on my custom board?

I assume that Xilinx have tried UG1186 v2019.2 OpenAMP Demo with a Xilinx development board, with the Linux project created from a Xilinx BSP, therefore is there a list of what PetaLinux tool config settings were set when Xilinx created the Xilinx BSP?...I could then look at them when setting the config settings for my Linux OS.

(I've also assumed that the UG1186 lists all the Device Tree nodes for the OpenAMP Demo that need to be added to system-user.dtsi).

Thank you.

[dave74321]

Hi @jovitac ,

Thanks for your help on this. 

lsmod returns

test_design login: root
Password:
root@test_design:~# modprobe rpmsg_char
root@test_design:~# lsmod
    Tainted: G
rpmsg_char 16384 0 - Live 0xffffff8000b97000
rpmsg_core 16384 1 rpmsg_char, Live 0xffffff8000b8e000
dmaproxy 16384 0 - Live 0xffffff8000b86000 (O)
al5d 16384 0 - Live 0xffffff8000b7d000 (O)
al5e 20480 0 - Live 0xffffff8000b61000 (O)
allegro 40960 2 al5d,al5e, Live 0xffffff8000b4f000 (O)
xlnx_vcu 16384 1 allegro, Live 0xffffff8000b3e000
xlnx_vcu_clk 20480 0 - Live 0xffffff8000b43000
zynqmp_r5_remoteproc 20480 0 - Live 0xffffff8000b34000
remoteproc 40960 1 zynqmp_r5_remoteproc, Live 0xffffff8000b23000
virtio_ring 20480 1 remoteproc, Live 0xffffff8000b19000
virtio 16384 1 remoteproc, Live 0xffffff8000b10000
xlnx_vcu_core 16384 4 - Live 0xffffff8000b08000
uio_pdrv_genirq 16384 0 - Live 0xffffff8000b00000

Hardware Description File (XSA file) attached.

Vivado diagram build TCL attached.

[jovitac]

I've tried this in qemu with the xsa you provided.

It seems like the driver you are missing is virtio_rpmsg_bus

root@forum-amp:/lib/firmware# echo image_echo_test > /sys/class/remoteproc/remoteproc0/firmware 
root@forum-amp:/lib/firmware# echo start > /sys/class/remoteproc/remoteproc0/state
[  123.630550] remoteproc remoteproc0: powering up r5@0
[  123.636923] remoteproc remoteproc0: Booting fw image image_echo_test, size 839768
[  123.708657]  r5@0: RPU boot from TCM.
[  123.726755] remoteproc remoteproc0: registered virtio0 (type 7)
[  123.728363] remoteproc remoteproc0: remote processor r5@0 is now up
Starting application...
root@forum-amp:/lib/firmware# Initialize remoteproc successfully.
creating remoteproc virtio
initializing rpmsg shared buffer pool
initializing rpmsg vdev
initializing rpmsg vdev
[  Try to create rpmsg endpoint.
123.817461] virtio_rpmsg_bus virtio0: rpmsg host is online
Successfully created rpmsg endpoint.
[  123.826169] xilinx-vcu xilinx-vcu: Could not get core_enc clock
[  123.828611] virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x0

root@forum-amp:/lib/firmware# lsmod
    Tainted: G  
virtio_rpmsg_bus 20480 0 - Live 0xffffff8000b60000
rpmsg_core 16384 1 virtio_rpmsg_bus, Live 0xffffff8000b57000
dmaproxy 16384 0 - Live 0xffffff8000b4f000 (O)
xlnx_vcu 16384 0 - Live 0xffffff8000b47000
xlnx_vcu_clk 20480 0 - Live 0xffffff8000b3e000
zynqmp_r5_remoteproc 20480 0 - Live 0xffffff8000b34000
remoteproc 40960 1 zynqmp_r5_remoteproc, Live 0xffffff8000b23000
virtio_ring 20480 2 virtio_rpmsg_bus,remoteproc, Live 0xffffff8000b19000
virtio 16384 2 virtio_rpmsg_bus,remoteproc, Live 0xffffff8000b10000
xlnx_vcu_core 16384 0 - Live 0xffffff8000b08000
uio_pdrv_genirq 16384 0 - Live 0xffffff8000b00000
root@forum-amp:/lib/firmware# 

[jovitac]

Your system-user.dtsi should be as below

/include/ "system-conf.dtsi"
/ {
};
/ {
	reserved-memory {
		#address-cells = <2>;
		#size-cells = <2>;
		ranges;
		rproc_0_dma: rproc@3ed400000 {
			no-map;
			compatible = "shared-dma-pool";
			reg = <0x0 0x3ed40000 0x0 0x100000>;
		};
		rproc_0_reserved: rproc@3ed000000 {
			no-map;
			reg = <0x0 0x3ed00000 0x0 0x40000>;
		};
	};

	zynqmp-rpu {
		compatible = "xlnx,zynqmp-r5-remoteproc-1.0";
		#address-cells = <2>;
		#size-cells = <2>;
		ranges;
		core_conf = "split";
		reg = <0x0 0xFF9A0000 0x0 0x10000>;
		r5_0: r5@0 {
			#address-cells = <2>;
			#size-cells = <2>;
			ranges;
			memory-region = <&rproc_0_reserved>, <&rproc_0_dma>;
			pnode-id = <0x7>;
			mboxes = <&ipi_mailbox_rpu0 0>, <&ipi_mailbox_rpu0 1>;
			mbox-names = "tx", "rx";
			tcm_0_a: tcm_0@0 {
				reg = <0x0 0xFFE00000 0x0 0x10000>;
				pnode-id = <0xf>;
			};
			tcm_0_b: tcm_0@1 {
				reg = <0x0 0xFFE20000 0x0 0x10000>;
				pnode-id = <0x10>;
			};
		};
	};


	zynqmp_ipi1 {
		compatible = "xlnx,zynqmp-ipi-mailbox";
		interrupt-parent = <&gic>;
		interrupts = <0 29 4>;
		xlnx,ipi-id = <7>;
		#address-cells = <1>;
		#size-cells = <1>;
		ranges;

		/* APU<->RPU0 IPI mailbox controller */
		ipi_mailbox_rpu0: mailbox@ff90000 {
			reg = <0xff990600 0x20>,
			      <0xff990620 0x20>,
			      <0xff9900c0 0x20>,
			      <0xff9900e0 0x20>;
			reg-names = "local_request_region",
				    "local_response_region",
				    "remote_request_region",
				    "remote_response_region";
			#mbox-cells = <1>;
			xlnx,ipi-id = <1>;
		};
	};
};

[dave74321]

Hi @jovitac,

Thank you for your reply.

To summarise in addition to that in UG1186 I have:

• Set DMA Contiguous Memory Allocator with size 256MB
• Used the dtsi you provided
• modprobe rpmsg_char
• modprobe virtio_rpmsg_bus

However the remoteproc still gets stuck - please see my attached boot log.

I need to have 1 processor running Linux and the other running FreeRTOS - OpenAMP/Remoteproc does not seem to work - is there any other (easy) way to do this?

[dave74321]

Hi @jovitac,

I would prefer to use Xilinx Zynq, as I have invested a lot of time in it.  So will struggle on for now trying to get more than one Zynq processor to work on my custom board.

Is there anything I should be trying?  e.g.

1. Use a different Vivado design
2. Use a different version of the tools (perhaps 2019.2 has a bug with RemoteProc?)
3. I have a Zedboard with Zynq-7000, so could try using 2 processors on that?

Could there be something wrong with power supplies that stops the R5 core booting up?

I'm running the PetaLinux tools on a virtual machine.  I assume that should be fine?

Thanks,

Dave

[jovitac · Accepted solution]

Hi Dave,

There are no bugs in remoteproc in 2019.2. It seems to be something in your design that is causing this to fail.

Do you have an evaluation board? Can you try this on a ZCU102. Start with a simple PS design only and then add PL components. See which one is causing the design to fail. It seems like maybe some memory region is overlapping.

Petalinux on virtual machine is fine. You can attach jtag cable and check the status of the R5 cores using xsdb.

[dave74321]

Hi @jovitac, Thank you for all your time on this.

As per one of your suggestions, I created a Vivado design that had a processing system IP but no FPGA logic.  I used the hardware description file (XSA) from this and image_echo_test worked fine!

I don't know what it was about my original Vivado test design HDF that seemed to mess things up (It was just a test design I created for trying the Video Codec Unit, VCU, in a Zynq UltraScale+), I will just discard my test design.

Thanks again, I learnt a few things along the way.

Dave