Enclustra Mars XU3 on EB1 with PetaLinux

[maerkl24]

Hello,

i want to run the OpenAMP Demo, as referred in UG1186 (Linux on A53 and Bare Metal on R5) on the Enclustra Mars XU3 board mounted on the EB1 base board.

But at first step by step.

What actually works, is to build and run the Linux provided by Enclustra https://github.com/enclustra-bsp/bsp-xilinx by booting form a SD Card.

My next step, i am actually stuck at, is to run a simple HelloWorld application on the linux build with enclustra tool.

I loaded a simple "HelloWorld.elf" file (build in SDK, C, Linux Application) into the rootfs under /bin but i couldn't find this file inside the embedded Linux. I removed the file extension and made it executable but this didn't help yet.

Nevertheless i think this problem is not important because i probably need to work with the petalinux tool because for the openAMP Demo. I need to change the device tree and i found no way to change the devicetree with the enclustra linux tool.

So i created a new petalinux project and specified the .hdf file from the enclustra reference design and built the project. But i couldn't run the linux on the board.

My question is, what is the simplest way to get the openAMP Demo running on the enclusta board?

Is it easier to use the linux provided by enclustra tool and change the devicetree (i have no idea how to do that) and build the openAMP Demo on SDK an load it onto the SD Card in the rootfs?

Or to use the petalinux tool and specifiy the openAMP Demo in the petalinux configuration (but i couldn't get a running linux build with petalinux tool yet)?

Sorry for that newbie question, but i am stock now for a few weeks and i have no idea how to continue.

Thanks in advance

[badegoke_f1]

Hi @maerkl24 ,
Did you get petalinux working on the XU3? I have an XU5 and there are no help materials to do this.

Thanks

Bade

[maerkl24]

Hey @badegoke_f1,

no, I couldn't get it working on the XU3, but I didn't try much because we changed the manufacturer.
But I'm sure it's possible to create a custom Linux for the Enclustra Boards, because they are doing the same. 

Useful documentes are:
- https://www.xilinx.com/support/documentation/sw_manuals/xilinx2019_2/ug1209-embedded-design-tutorial.pdf (pp. 14 - 19): Shows how to create a basic Vivado Design (the tutorial is for the Xilinx ZCU102 Board, but it's the same for any other board, except that you have to fill in some config manually, but i think there is a Vivado reference design from Enclustra)
- https://www.xilinx.com/support/documentation/sw_manuals/xilinx2019_2/ug1144-petalinux-tools-reference-guide.pdf (pp. 20 -21): Shows how to create a empty PetaLinux project. (pp. 23 - 24) Shows how to import the hardware description form Vivado.

I think you also have to modify the Device-Tree (system-user.dtsi). I think you will find the required information somewhere in the Enclustra Build Tool, because there are also some source files for the building of the embedded Linux. 

I hope this will help you.
Tell me if you got it working. ;-)

Kilian

[badegoke_f1]

Hi @maerkl24 ,

I was able to get the XU5 and PE1+ carrier working with petalinux booting from SDcard. I had to modify the device-tree and use petalinux-config to configure the sd_port used as well as change the boot image configuration.
I also noticed my clocks for my PL side IPs had to be up and running before boot otherwise it stalls.

Thanks 

Bade

[pmoschov]

Dear @badegoke_f1, in that case could you provide some information on what was finally needed to make the leap to Petalinux?

Do you need at all the Enclustra building environment to get some of the files, or you could do everything directly from the reference design?

Best,
bakalismas

[badegoke_f1]

Hi @pmoschov ,

I didn't really use outputs from the Enclustra Build Environment. Although I did initially boot up with that.

For Petalinux, your best bet is

1. Using the Reference design as a starting point. Note: make sure your PS configurations(Memory, clocks, I/Os are all set correctly as per your Module of choice).
2. Export the generated XSA to petalinux and generate a project using Xilinx templates depending on your SoM device(Zynq or ZynqMP).
3. Import the XSA using petalinux build commands.
4. Use petalinux Config command to set things like boot device, ethernet, enable uart and memory sd or where rootfs location is.
5. Modify the device tree using "system-user.dtsi" to match the configurations in petalinux-config. In my case I had something like:

/include/ "system-conf.dtsi"
/ {
chosen {
		bootargs = "earlycon console=ttyPS0,115200 clk_ignore_unused cma=1000M cpuidle.off=1 root=/dev/mmcblk0p2 rw rootwait";
		stdout-path = "serial0:115200n8";
	};
};
&sdhci1 {
	no-1-8-v;
	disable-wp;
};

&sdhci0 {
	status = "disabled";
};

Note: I had to disable the WP, I found it in some forum online.

•  Modify the kernel depending on your included IPs, or packages you need using petalinux kernel config. I noticed, that boot would stall if some drivers or packages were not enabled. I think this is depending on your hardware imported from Vivado.
• Generate/package boot images depending on your boot choice using petalinux commands.
• You should be ready to boot petalinux on the Enclustra SoM.

The petalinux user guide will be very helpful. 

Regards

Bade

[pmoschov]

Hello @badegoke_f1 ,

that was really helpful! Thanks a lot.

Did you try to use the device tree as in Enclustra xilinx-linux fork, or you had to make it from scratch?

Kind regards,

Paris

[badegoke_f1]

Hi @pmoschov ,

Its a good point you mentioned that. In hind sight, maybe I should have done that I might have made things easier I suppose. I built mine up from scratch by :

- Decompiling the default generated dtb from the XSA.

- Modifying with the required functions/nodes I want to enable with an overlay in "system-user.dts"

- Recompiling the device tree.

Kindly share your findings if you can. I can learn a thing or two and hopefully it helps someone else.

Best regards

Bade

Bade

Regards

Bade

[KentoOkazaki]

@badegoke_f1 

Hello, I am using Mercury XU5 and PE1 and want to boot Linux from SD card too.

I have exported an HDF (because I used Vivado 2019.1) based on the Reference Design by Enclustra.

I imported the HDF to PetaLinux 2018.3 as below: 

$ petalinux-create --type project --template zynqMP --name XU5_5EV
$ cd XU5_5EV
$ # (I copied MercuryXU5_PE1.sdk which contains system_top.hdf in this directory 'XU5_5EV')
$ petalinux-config --get-hw-description=MercuryXU5_PE1.sdk

But I failed to boot. Could you possibly tell me how you did petalinux-config for your booting?  Why did you use XSA instead of HDF? I want to know what you have done in detail, especially the step of overwriting the system-user.dtsi. Thank you very much in advance!

P.S. I was successful to boot with EBE, but have no idea how to boot with PetaLinux.

[badegoke_f1]

Hi @KentoOkazaki ,

A couple of things...

1. I didn't use use the MecuryXU5_PE1.sdk in my Petalinux build, If I understand you correctly. I created a petalinux project as you have done using a zynqMP template. After that, I imported the XSA file I generated from the Enclustra Vivado Reference design( Although slightly I modified for the DDR size on my SoM).Set the appropriate config in the project,kernel and rootfs.
2. Be sure to verify the SD card boot selection is correct in the petalinux config settings. I had to modify one in my system-user.dtsi and also change the boot selection in the petalinux-config GUI.I think the PE1 SD is &sdhci1.
3. I would suggest you use the same Petalinux and Vivado version. I have had issues in the past with using a different vivado and petalinux version.

Can share your system-user.dtsi and also a decompiled system.dtb ?

Regards

Bade

[KentoOkazaki]

Dear @badegoke_f1 

Thank you very much for your reply! I have several questions for you...

- Did you configure "DTG Settings ---> (template) MACHINE_NAME"? My boot log says 

U-Boot 2018.01 (Nov 08 2020 - 08:39:37 +0000) Xilinx ZynqMP ZCU102 rev1.0

although I don't use ZCU102.

- In which step did you edit the system-user.dtsi? Is it before running "petalinux-build" command?

- Can I ask you what versions of PetaLinux and Vivado you used? I used Vivado 2019.1 because Enclustra Support said that the Reference Design supports 2019.1.

- I'm new to something like this development and didn't know what I should write in system-user.dtsi. So I just copied your system-user.dtsi. 

I attach my system.dts (as a text file), and my latest boot log which booting was failed. I thank you very much for your helping.

[badegoke_f1]

Hi @KentoOkazaki ,

Answering your questions: 

- Did you configure "DTG Settings ---> (template) MACHINE_NAME"? My boot log says 

No I didn't change mine. Its not overly important in this case of yours not booting.

-- In which step did you edit the system-user.dtsi? Is it before running "petalinux-build" command?

Yes I edit the system-user.dtsi before running the "petalinux-build" command.

- Can I ask you what versions of PetaLinux and Vivado you used? I used Vivado 2019.1 because Enclustra Support said that the Reference Design supports 2019.1.

I have used 2019.1, 2019.2 and currently on 2020.1. I have it booting on all 3 versions.

Back to your boot issue: 

If you have copied my system-user.dtsi, you might have to make some changes in the petalinux-config boot settings. However I will suggest we take it one at a time. 

Going through your boot logs, it seems its your mmc not configured correctly. Can you run the following command "mmcinfo" and "Zynq> mmc list" in the ZynqMP> stall location on boot.

It will tell if the device sees your SD card at mmc@ff170000. It should tell you the SD card information.  This is mine:

ZynqMP> mmcinfo
Device: mmc@ff170000
Manufacturer ID: 27
OEM: 5048
Name: SD32G
Bus Speed: 49995000
Mode: SD High Speed (50MHz)
Rd Block Len: 512
SD version 3.0
High Capacity: Yes
Capacity: 29 GiB
Bus Width: 4-bit
Erase Group Size: 512 Bytes
ZynqMP>

If it does not show up, We can then go through your petalinux-config settings or try to use mmc cmd to set the right device.

Best regard

[KentoOkazaki]

Dear @badegoke_f1 

Thank you for answering my questions.

My outputs of mmcinfo are below:

ZynqMP> mmcinfo
Device: mmc@ff170000
Manufacturer ID: 3
OEM: 5344
Name: SL16G
Tran Speed: 50000000
Rd Block Len: 512
SD version 3.0
High Capacity: Yes
Capacity: 14.8 GiB
Bus Width: 4-bit
Erase Group Size: 512 Bytes
ZynqMP>

I want to put rootfs in the 2nd partition of SD card, and "BOOT.BIN", "image.ub", and "system.dtb (I petalinux-configured to read the device tree outside on image.ub)" in the 1st partition. Just like this: https://enclustra.github.io/ebe-docs/user-doc-xilinx/index_xilinx.html#sd-card-mmc

Does the Bad device mmc 1 mean the 2nd partition? If so, could you tell me what I must do?

By the way I configured the bootargs as "DTG Settings ---> Kernel Bootargs ---> [] generate boot args automatically (console=ttyPS0,115200 root=/dev/mmcblk0p2 rw earlyprintk rootfstype=ext4 rootwait devtmpfs.mount=1)". And as of my SD card, it doesn't have any problem I guess, because I succeeded in booting Linux with EBE.

[badegoke_f1]

Hi @KentoOkazaki ,

I think your board stalls after FSBL as it is looking in the wrong place to load the boot images.

In your Vivado project verify that sd1/emmc is enabled to MIO46-51 in the PS. This is shared via a Mux on the PE1 carrier board between the eMMC Managed NAND Flash and Micro SD-Card Slot(our intended boot medium).

Using petalinux-config command, change the following:

1. Change the sdio config to psu_sd_1 in Subsystem AUTO Hardware Settings --> SD/SDIO Settings --> Primary SD/SDIO (psu_sd_1)
2. Under Advanced bootable images storage Settings --> image storage media(primary sd)
                                                                                      -->    u-boot env partition settings --> primary_sd
                                                                                      -->    kernel image settings --> primary_sd
                                                                                      -->    dtb image settings --> primary_sd
3. Change Image Packaging Configuration --> Root filesystem type (EXT4 (SD/eMMC/SATA/USB))
                                                                      -->  (/dev/mmcblk0p2) Device node of SD
4. Modify the system-user.dtsi which I assume you have done from my previous posted example. The above changes are based on MY system-user.dtsi.
5. Optionally configure rootfs and kernel with "petalinux-config -c kernel"...otherwise run "petalinux-build" command.
6. Generate the boot images using petalinux-package command in the /Image/linux directory(refer to Xilinx documentation)
7. Prepare a boot SD card with two partitions named BOOT(FAT32 format) and rootfs(EXT4 format).
8. Copy Image.ub, BOOT.bin, boot.scr and system.dtb to the FAT32 partition of the SD card.
9. Extract rootfs.tar.gz to the EXT4 partition using "sudo tar -xvf rootfs.tar.gz -C <SD card rootfs EXT4 partition locaton>"
10. Insert the SD card in the PE1 carrier.

Your board should boot as per your requirement of having separate rootfs from image.ub. I hope this helps. 

Please note that the petalinux configuration settings and naming might be slightly different as I am using 2020.1.

Regards

Bade

I hope this helps

[KentoOkazaki]

Hello @badegoke_f1 

I appreciate your detailed advice.

I installed Vivado 2020.1, and exported XSA file. 

I also installed PetaLinux 2020.1. Here's what I did...

$ petalinux-create --type project --template zynqMP --name XU5_5EV

$ cd XU5_5EV

$ (copy system_top.xsa to here)

$ petalinux-config --get-hw-description=./

Subsystem AUTO Hardware Settings --> SD/SDIO Settings --> Primary SD/SDIO (psu_sd_1)

Under Advanced bootable images storage Settings --> image storage media(primary sd)
                                                                                   -->    u-boot env partition settings --> primary_sd
                                                                                   -->    kernel image settings --> primary_sd
                                                                                   -->    dtb image settings --> primary_sdChange Image Packaging Configuration --> Root filesystem type (EXT4 (SD/eMMC/SATA/USB))
                                                                   -->  (/dev/mmcblk0p2) Device node of SD

I kept other settings default.

$ vi project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi (copied your file)

$ petalinux-build

$ cd images/linux

$ petalinux-package --boot --fsbl zynqmp_fsbl.elf --u-boot u-boot.elf --atf bl31.elf --pmufw pmufw.elf

$ (I copied BOOT.BIN, image.ub, system.dtb, and boot.scr to 1st partition, and rootfs to 2nd partition)

But my booting stalls shortly after Xilinx Zynq MP First Stage Boot Loader. Release 2020.1 Nov 20 2020 - 10:54:34 appears. Do I miss something? 

[badegoke_f1]

Hi @KentoOkazaki ,

Sorry I was away for some days. Kindly post your boot logs.

Thanks

[KentoOkazaki]

Hello @badegoke_f1 ,

I petalinux-configured as below

- Subsystem AUTO Hardware Settings ---> SD/SDIO Settings ---> Primary SD/SDIO (psu_sd_1)
- Subsystem AUTO Hardware Settings ---> Advanced bootable images storage Settings ---> boot image settings ---> image storage media (primary sd)
- Subsystem AUTO Hardware Settings ---> Advanced bootable images storage Settings ---> u-boot env partition settings ---> image storage media (primary sd)
- Subsystem AUTO Hardware Settings ---> Advanced bootable images storage Settings ---> kernel image settings ---> image storage media (primary sd)
- Subsystem AUTO Hardware Settings ---> Advanced bootable images storage Settings ---> jffs2 rootfs image settings ---> image storage media (primary flash)
- Subsystem AUTO Hardware Settings ---> Advanced bootable images storage Settings ---> dtb image settings ---> image storage media (primary sd)
- DTG Settings ---> Kernel Bootargs ---> [*] generate boot args automatically [*] enable kernel earlyprintk (earlycon console=ttyPS0,115200 clk_ignore_unused root=/dev/mmcblk0p2 rw rootwait)
- Image Packaging Configuration ---> Root filesystem type (EXT4 (SD/eMMC/SATA/USB))

and copied your system-user.dtsi

/include/ "system-conf.dtsi"
/ {
    chosen {
	    	bootargs = "earlycon console=ttyPS0,115200 clk_ignore_unused cma=1000M cpuidle.off=1 root=/dev/mmcblk0p2 rw rootwait";
		    stdout-path = "serial0:115200n8";
	    };
    };
# 
&sdhci1 {
	no-1-8-v;
	disable-wp;
};
# 
&sdhci0 {
	status = "disabled";
};

And I've got the bootlog attached to this response. It seems that a block of the same logs comes up every 60 seconds, and I cannot reach the Linux. 

Could you possibly tell me what the sentences mean in system-user.dtsi? Thank you. 

[badegoke_f1]

Hi @KentoOkazaki ,

Looking through your boot logs I can see it is booting up as the kernel starts which means it successfully located the boot image and also rootfs in the sdcard .. I can also see the different CPU cores start which is some progress. 

However at [ 30.930709] the CPU stalls. I have tried to look for errors in the boot log which  cannot really see. Can you confirm what BSP you have used to generate the Petalinux project?

I will suggest using a bare ZynqMp Template and import your .XSA from your built Enclustra Base project. Use the same config and device-tree you have used in this project. I see some boot log prompts which I haven't noticed on mine. I do not know if you have some kernel configs different. It is always best to start with the bare minimum.

I used the ZynqMP template as my base Petalinux project... then I kept modifying the device tree, kernel and dtb as I added more functionality in PS and PL and imported the updated XSA to the project.

Kindly let me know how it goes.

Cheers
Bade

P.S 
The node &sdhci1 disables write protect and indicates the voltage configuration for the sdcard. We disable scared 0
r00t=...points to the partition of the rootfs, cpuidle=1 disables cpu idle. 
you can look them up for more information.

[KentoOkazaki]

Hello @badegoke_f1. I appreciate your response. 

What do you mean by "BSP"? If you mean it's a template when creating a PetaLinux project, I used zynqMP like petalinux-create --type project --template zynqMP --name <project_name>. 

I don't think that what I did is too different from what you did... 

• Vivado 2020.1 and PetaLinux 2020.1
• Used XU5 Reference Design by Enclustra
• In Tcl Console in Vivado window, run source scripts/tcl_settings.sh
• Checked sd1 is enabled in MIO 46-51
• Exported XSA (system_top.xsa) without any changes in hardware design
• Copied system_top.xsa to the home directory in my Ubuntu and run petalinux-config --get-hw-description=~/
• Edited system-user.dtsi and petalinux-build
• petalinux-packaged and copied BOOT.BIN, image.ub, system.dtb and boot.scr to 1st partition

Looking for solutions to my trouble on the Internet, I may have to make some changes in petalinux-config -c kernel. I'm trying on it.

Thank you for your advice anyway.

[badegoke_f1]

Hi @KentoOkazaki ,

Any luck with the boot?

Regards

[KentoOkazaki]

Dear @badegoke_f1 

NO... I have tried to disable CONFIG_CPU_IDLE and CONFIG_EDAC_CORTEX_ARM64 but don't have any progress. 

I'm referring to these pages

• https://www.xilinx.com/support/answers/69433.html
• https://forums.xilinx.com/t5/Embedded-Linux/zynqmp-rcu-INFO-rcu-sched-detected-stalls-on-CPUs-tasks/td-p/939934