cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Visitor
Visitor
3,669 Views
Registered: ‎08-29-2017

U-BOOT Networkboot - loading *.elf file for another core

Hi all,

We're using XAPP 1078 AMP configuration. Core 0 is running Petalinux and Core 1 is bare metal.

https://www.xilinx.com/support/documentation/application_notes/xapp1078-amp-linux-bare-metal.pdf 

 

With this configuration, we'd like to network boot both cores. By using U-Boot and TFTP/BOOTP, we are able to properly boot the Petalinux on Core0 from the Network. The problem is with loading the *.elf file for the other core (CORE1) in the same u-boot script. We'd like to load the *.elf with some kind of elf loader and then continue booting Linux which kicks off the baremetal afterwards. 

 

We tried the u-boots bootelf command, but it seems to boot CORE 0 with our *.elf that is supposed to be for CORE1.

 

Could you please help us to find  way to load *.elf in u-boot or to network boot the 2 cores running separately?

 

We'd like to avoid 2-stage booting ( start - load new files - restart ). We're using 2016.3 tools for all our developments. 

 

Thanks in advance for any valuable information. 

0 Kudos
13 Replies
Highlighted
Voyager
Voyager
3,648 Views
Registered: ‎06-24-2013

Re: U-BOOT Networkboot - loading *.elf file for another core

Hey @oleniuk,

 

The problem is with loading the *.elf file for the other core in the same u-boot script ...

Why not set aside some memory for the bare metal core (you probably do that already), load the elf into this memory range via tftp (just request a different file or with a different id), then boot linux and kick off the bare-metal core from there?

 

Best,

Herbert

-------------- Yes, I do this for fun!
0 Kudos
Visitor
Visitor
3,643 Views
Registered: ‎08-29-2017

Re: U-BOOT Networkboot - loading *.elf file for another core

Hi @hpoetzl,

Thank you for fast reply!

The problem is that we cannot just load the *.elf in the memory through u-boot. As far as we understand, *.elf is not a "true binary" file, it contains a header and blobs of binary code that should be placed into different places, hence it should be loaded through some kind of elf parser.

 

We tried TFTPing the *.elf in the [ADDR] and than starting our CORE1 from the [ADDR] and it does not work because of the reason explained above. 

 

Normally the *.elf file is loaded in FSBL which does not have the network capabilities. Therefore we'd like to load the core1 *.elf from U-BOOT.

 

Please let us know if you see any solution, 

0 Kudos
Highlighted
Voyager
Voyager
3,629 Views
Registered: ‎06-24-2013

Re: U-BOOT Networkboot - loading *.elf file for another core

Hey @oleniuk,

 

Thank you for fast reply!

You're welcome!

 

... hence it should be loaded through some kind of elf parser.

At the point where you are dealing with the other core, you already have Linux running and you can easily parse and distribute the elf in preparation for the bare metal core start.

 

Another option, in case that is too late for you (think other core should boot early) you can reuse the FSBL part which handles the elf for the second core and load a special FSBL plus the elf via TFTP.

 

There are really a number of ways to deal with this, depending on your requirements.

 

Hope this clarifies,

Herbert

-------------- Yes, I do this for fun!
0 Kudos
Highlighted
3,614 Views
Registered: ‎08-29-2017

Re: U-BOOT Networkboot - loading *.elf file for another core

Hi @hpoetzl,

Thanks again for your suggestions, I just had a few follow-up questions:

 

Can you clarify some of the mechanics of your second option, i.e., using a second instance of FSBL on Core 1? Are you suggesting we create a second FSBL that sets up Core 1 to launch the baremetal ELF in parallel (so to speak), while Core 0 is booting Petalinux? 

 

"At the point where you are dealing with the other core, you already have Linux running and you can easily parse and distribute the elf in preparation for the bare metal core start."

As for this, I believe we would run into a similar problem as described here. Won't the elf parser not be able to actually load to the addresses described in the elf program and section headers, since it's running on half the memory? 

 

Bharat

0 Kudos
Highlighted
Voyager
Voyager
3,606 Views
Registered: ‎06-24-2013

Re: U-BOOT Networkboot - loading *.elf file for another core

Hey @bharadwaj.srinivasan,

 

Are you suggesting we create a second FSBL that sets up Core 1 to launch the baremetal ELF in parallel (so to speak), while Core 0 is booting Petalinux?

Yes, except that it wouldn't be a full fledged FSBL but more an elf unpacker.

 

Won't the elf parser not be able to actually load to the addresses described in the elf program and section headers, since it's running on half the memory?

Just because a memory region is not used by Linux doesn't necessarily mean that Linux doesn't have access to it. This really depends on the configuration.

 

Hope this clarifies,

Herbert

-------------- Yes, I do this for fun!
0 Kudos
Highlighted
Visitor
Visitor
3,560 Views
Registered: ‎04-03-2017

Re: U-BOOT Networkboot - loading *.elf file for another core

Hi Herbert,

 

Another option, in case that is too late for you (think other core should boot early) you can reuse the FSBL part which handles the elf for the second core and load a special FSBL plus the elf via TFTP

This sounds reasonable, essentially you could have a second boot stage where this minimal FSBL parses/loads the bare metal elf into 0x30000000 and puts core 1 into the WFE loop. I imagine you would wrap this FSBL and bare metal elf up with bootgen to be delivered. But how do you kick this second FSBL stage off from uboot? And then how would you return to core 0 to load the kernel and kick off Linux? From ug821 and ug585, it's not exactly clear (maybe I'm missing something).

 

At the point where you are dealing with the other core, you already have Linux running and you can easily parse and distribute the elf in preparation for the bare metal core start.

How easy is this, really? Of course, the ELF format is something you can look up, and implement a custom application in Linux to parse the ELF file, possibly mmap /dev/mem and dump the executable binary contents (without the header) into memory starting at 0x3000000. That seems like a very manual process, prone to error, and a maintenance burden.

 

There are really a number of ways to deal with this, depending on your requirements

This follows the AMP approach of XAPP 1078: upon boot, the bitstream is loaded into PL, the bare metal application code is loaded in 0x30000000, core 1 starts in the WFE loop at 0xFFFFFFF0, and the PetaLinux kernel gets kicked off and the ramdisk is loaded. A Linux application then starts the bare metal application by writing 0x30000000 to 0xFFFFFFF0. 

To avoid having to flash artifacts onto the board during a "manufacturing" step, it would be great to network boot the artifacts, having only to flash the FSBL and Uboot image. Uboot then takes care of the rest, getting/loading the bitsream, bare metal application (this is apparently the hard part!), and kernel/ramdisk images via TFTP. Not sure if anyone else has tried net booting for AMP on Zynq. 

 

0 Kudos
Highlighted
Voyager
Voyager
3,549 Views
Registered: ‎06-24-2013

Re: U-BOOT Networkboot - loading *.elf file for another core

Hey @maxx.becker,

 

Okay, I try to answer all the questions in one reply, by explaining a little how the boot works and what's already there ...

 

When the Zynq PS is reset, it ends up executing the bootrom, which, among other stuff, sets up the Wait for Event (WfE) loop in the On Chip Memory (OCM) somewhere above 0xFFFFFF00. This is basically a loop which constantly jumps to the address specified in 0xFFFFFF00, which by default is the start of said loop. It is very likely that this code is already executed by CPU1, but we know for sure that CPU1 ends up in this loop where it will spin waiting for a change in the jump address. CPU0 on the other hand will read the mode registers and do some basic initialization and finally start booting the FSBL from the specified device (based on the mode).

 

Now this is the first place where we can hook in and do some crazy stuff like for example loading code for both CPUs, but because we don't want to write everything from scratch, we go for a known-good solution here called U-Boot, a proven bootloader which, when wrapped into the FSBL code provided by Xilinx will be able to download data via TFTP into memory. Note that this in itself is already enough to boot both CPUs with separate binaries loaded via network, by simply loading the binary blobs at different addresses and then setting the jump address for the second CPU to the address of the second binary and branching off to the address of the first binary.

 

Now let's say we don't want to load binary blobs but instead use ELF files to populate the memory. The first option here is a very basic elf loader already present in U-Boot which probably can be used with only small modifications to unpack an ELF file without booting it. Once the memory is populated, booting CPU1 is the same as before and U-Boot can continue with the boot process for CPU0. Of course, instead of unpacking the ELF in U-Boot, we could load a full fledged ELF unpacker into memory, pass the base address of the ELF file to it and let CPU1 crunch away on this task while again continuing with our boot on CPU0.

 

Finally, if we don't mind the delay or maybe even desire to load the code for the second CPU under Linux, we can do so by reserving some memory and using userspace tools to populate it with unpacked ELF data or with the beforementioned ELF unpacker and kick of the second CPU from Linux the same way we would do it from U-Boot.

 

Hope this clarifies,

Herbert

-------------- Yes, I do this for fun!
0 Kudos
Highlighted
Visitor
Visitor
3,536 Views
Registered: ‎04-03-2017

Re: U-BOOT Networkboot - loading *.elf file for another core

When the Zynq PS is reset, it ends up executing the bootrom, which, among other stuff, sets up the Wait for Event (WfE) loop in the On Chip Memory (OCM) somewhere above 0xFFFFFF00. This is basically a loop which constantly jumps to the address specified in 0xFFFFFF00, which by default is the start of said loop. It is very likely that this code is already executed by CPU1, but we know for sure that CPU1 ends up in this loop where it will spin waiting for a change in the jump address. CPU0 on the other hand will read the mode registers and do some basic initialization and finally start booting the FSBL from the specified device (based on the mode).

Understood, as explained in Xilinx reference materials.

 

Now let's say we don't want to load binary blobs but instead use ELF files to populate the memory.

This would be preferred, as opposed to your second paragraph response about loading the raw bare metal application binary blob and having CPU1 start executing it, because you can use the Xilinx tools to prepare your bare metal application (along with encapsulating all the other configuration information, e.g. what's in ldscript). Additionally, you may not want CPU1 to start executing immediately if you're following the XAPP 1078 approach, and you want the linux application to set up shared resources, etc. before. I understand you can write any value to arbitrary memory locations in u-boot, but consider the situation where you want CPU1 to continue the WFE loop until a Linux application kicks it off.

 

 we can do so by reserving some memory and using userspace tools to populate it with unpacked ELF data or with the beforementioned ELF unpacker 

As I mentioned before, this is valid, but not preferred:  Of course, the ELF format is something you can look up, and implement a custom application in Linux to parse the ELF file, possibly mmap /dev/mem and dump the executable binary contents (without the header) into memory starting at 0x3000000. That seems like a very manual process, prone to error, and a maintenance burden.

 

Essentially, it's asked to be able to load an elf from uboot. This isn't built in. So the most reasonable options, based on your responses, are to either:

1. Use u-boot to grab a new .bin with a second fsbl and bare metal elf. Somehow kick this second boot loader off, possibly with some multiboot feature (?). This 2nd FSBL will load the bare metal application where necessary, then hopefully (?) you pick up where the 1st FSBL left off and CPU0 continues to starting the kernel. CPU1 is still in the WFE event loop and the bare metal application gets kicked off via the linux application. 

2. Modify uboot to load the elf and not reboot (possibly modify the bootelf command). 

 

The mechanics of how these options are implemented is not clear. 

0 Kudos
Highlighted
Voyager
Voyager
3,532 Views
Registered: ‎06-24-2013

Re: U-BOOT Networkboot - loading *.elf file for another core

Hey @maxx.becker,

 

I can't follow your reasoning and I don't see why you build so complicated constructs.

 

Given that you don't want to unpack the ELF under Linux (for whatever reason), I see the following options ...

  1. Modify U-Boot to execute bootelf without the boot part (probably rather simple).
  2. Create or adapt an ELF unpacker and use it as BL for the second CPU.
  3. Write your own FSBL code which can TFTP boot both cores.

There is no problem with early or late start for CPU1, you can do it as soon as you've loaded the data into memory or at some later point in the Linux kernel or userspace.

 

Best,

Herbert

-------------- Yes, I do this for fun!
0 Kudos
Highlighted
Visitor
Visitor
2,329 Views
Registered: ‎08-29-2017

Re: U-BOOT Networkboot - loading *.elf file for another core

Hi @hpoetzl,

Thanks for your explanations. I changed u-boots "bootelf" function so that it only loads the *.elf and does not execute it. After analyzing the code, I concluded that I don't need to recompile, I just need to set the "autostart" environmental variable to "no". 

 

The u-boot source code for bootelf is here:
https://github.com/Xilinx/u-boot-xlnx/blob/master/cmd/elf.c 
(see line 162 and 190, function "do_bootelf")

 

In U-Boot, I tried loading different *.elf files and this approach is not working properly. This is what I did:
1. bootp                      ( assigns the IP and takes the TFTP server ip)

2. tftpboot 0x0 BM.elf (also tried addresses 0xC, 0x1000, 0x100000)

3. setenv autostart no (make bootelf only load the lf, without restart)

4. bootelf 0x0              (also tried flags -s and -p)

5. I get "Cache misalignment" warning (not sure if relevant). 

Zynq> bootelf 0x0
CACHE: Misaligned operation at range [30000000, 30004434]
CACHE: Misaligned operation at range [30004434, 3000444c]
CACHE: Misaligned operation at range [3000444c, 30004464]
CACHE: Misaligned operation at range [30004464, 300047f4]
CACHE: Misaligned operation at range [300047f8, 300050e4]
CACHE: Misaligned operation at range [300050e4, 300050e8]
CACHE: Misaligned operation at range [3000c000, 3000c008]
CACHE: Misaligned operation at range [3000c008, 3000c00c]
CACHE: Misaligned operation at range [3000c00c, 3000c0f0]
CACHE: Misaligned operation at range [3000c0f0, 3000e0f0]
CACHE: Misaligned operation at range [3000e0f0, 300118f0]
Zynq>

 

6. I can see that the *.elf is loading to the address 0x30000000 and the board is not rebooting afterwards (that's what I wanted).

I continue booting Linux. 


7.Then, when trying to start the CORE1 in Linux, (set the start address 0x30000000 to WFE loop)

 

poke 0xFFFFFFF0 0x30000000

 

I can observe that the processor is not running (I have a shared OCM register to observe changing heartbeat).

 

Sometimes I get Linux kernel panic like this :

 

root@zynq:~#
root@zynq:~#poke 0xfffffff0 0x30000000
Modules linked in: ipv6 CPU: 0 PID: 661 Comm: mmcqd/0 Not tainted 4.6.0-xilinx #2 Hardware name: Xilinx Zynq Platform [<c010e48c>] (unwind_backtrace) from [<c010a6b0>] (show_stack+0x10/0x14) [<c010a6b0>] (show_stack) from [<c02d05b4>] (dump_stack+0x80/0x9c) [<c02d05b4>] (dump_stack) from [<c01362a4>] (__schedule_bug+0x44/0x60) [<c01362a4>] (__schedule_bug) from [<c05a65d0>] (__schedule+0x78/0x458) [<c05a65d0>] (__schedule) from [<c05a6a60>] (schedule+0xb0/0xcc) [<c05a6a60>] (schedule) from [<c04900e0>] (mmc_start_req+0xb4/0x328) [<c04900e0>] (mmc_start_req) from [<c049c554>] (mmc_blk_issue_rw_rq+0x408/0x9d0) [<c049c554>] (mmc_blk_issue_rw_rq) from [<c049cef8>] (mmc_blk_issue_rq+0x3dc/0x448) [<c049cef8>] (mmc_blk_issue_rq) from [<c049e454>] (mmc_queue_thread+0xb8/0x150) [<c049e454>] (mmc_queue_thread) from [<c013278c>] (kthread+0xd4/0xec) [<c013278c>] (kthread) from [<c0106f78>] (ret_from_fork+0x14/0x3c) Unable to handle kernel NULL pointer dereference at virtual address 00000028 pgd = c0004000 [00000028] *pgd=00000000 Internal error: Oops - BUG: 17 [#1] PREEMPT SMP ARM Modules linked in: ipv6 CPU: 0 PID: 661 Comm: mmcqd/0 Tainted: G W 4.6.0-xilinx #2 Hardware name: Xilinx Zynq Platform task: ee6627c0 ti: df8f2000 task.ti: df8f2000 PC is at __schedule+0x25c/0x458 LR is at __schedule+0x1dc/0x458 pc : [<c05a67b4>] lr : [<c05a6734>] psr: 600f0193 sp : df8f3e08 ip : 00000000 fp : df8f3e3c r10: 00000000 r9 : c1402e1c r8 : 00000000 r7 : 00000000 r6 : ee46e600 r5 : ee9cd2c0 r4 : ee6627c0 r3 : 00000028 r2 : 000052d9 r1 : df8f2000 r0 : 00000000 Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none Control: 18c5387d Table: 1fa0004a DAC: 00000051 Process mmcqd/0 (pid: 661, stack limit = 0xdf8f2210) Stack: (0xdf8f3e08 to 0xdf8f4000) 3e00: 00000000 df8a3f80 00000001 c049f314 00000000 df8f2000 3e20: df8d5660 df8d5540 df8f3e5c df8f3ed4 df8d5434 df8f3e68 df8f3e4c c05a6a60 3e40: df889800 df8d5660 df889b10 c04900e0 df8d5550 df8ac800 df87d1e0 00000000 3e60: ee6627c0 c0149b4c df8f3e68 df8f3e68 00000000 df8d5550 df8d5408 df8ac800 3e80: df87d1e0 df87d1e0 df889800 df8d5400 00000000 c049c554 ee4e1180 df8d5400 3ea0: 00000000 00000000 00000000 00000000 00000000 df8d5554 00000000 00000000 3ec0: f8f00100 c1402f6c df8f3ef0 c1411d80 f8f01100 c01013dc c05a9f2c df87d1e0 3ee0: df8ac800 df8d5408 df8d5410 ee795530 df889800 df8d5400 df8d5400 c049cef8 3f00: df8d5408 ffffe000 24590000 df8d5410 ee795530 df8f3f40 df8f2000 df8d5408 3f20: ffffe000 df87d1e0 df8d5410 ee795530 df8f3f40 df8f2000 24590000 c049e454 3f40: ee6627c0 df8ad340 00000000 df8d5408 c049e39c 00000000 00000000 00000000 3f60: 00000000 c013278c df8f3f7c 00000000 ee61ec80 df8d5408 00000000 00000000 3f80: df8f3f80 df8f3f80 00000000 00000000 df8f3f90 df8f3f90 df8ad340 df8ad340 3fa0: c01326b8 00000000 00000000 c0106f78 00000000 00000000 00000000 00000000 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 e7fddef0 e7fddef0 [<c05a67b4>] (__schedule) from [<c05a6a60>] (schedule+0xb0/0xcc) [<c05a6a60>] (schedule) from [<c04900e0>] (mmc_start_req+0xb4/0x328) [<c04900e0>] (mmc_start_req) from [<c049c554>] (mmc_blk_issue_rw_rq+0x408/0x9d0) [<c049c554>] (mmc_blk_issue_rw_rq) from [<c049cef8>] (mmc_blk_issue_rq+0x3dc/0x448) [<c049cef8>] (mmc_blk_issue_rq) from [<c049e454>] (mmc_queue_thread+0xb8/0x150) [<c049e454>] (mmc_queue_thread) from [<c013278c>] (kthread+0xd4/0xec) [<c013278c>] (kthread) from [<c0106f78>] (ret_from_fork+0x14/0x3c) Code: 1a000008 e5867244 e2873028 f593f000 (e1932f9f) ---[ end trace 38245981a3c1d3eb ]--- note: mmcqd/0[661] exited with preempt_count 2

 

 

If I place the same BM.elf in BOOT.BIN, then it's the FSBL that loads the BM.elf, the operation is successful and I can start the CORE1 in Linux and I see the CORE1 heartbeat.

 

I compared the memory in 0x30000000 - 0x30000010 in the "FSBL load" working case and in the "U-Boot load" not working case and they look exactly the same (I didn't compare all the memmory content though).

 

Hence, I conclude this is a CORE1 configuration or memory configuration problem. When the FSBL loads the CORE1 BM.elf I image it also kind of set's it up. U-boots bootelf cmd might not do it since it's designed for loading elf for CORE0 which is already running, not the other core.

 

I read in the documentation of XAPP 1079 (2cores bare metal) :
https://www.xilinx.com/support/documentation/application_notes/xapp1079-amp-bare-metal-cortex-a9.pdf

897B330B.PNG

Maybe I should send the SEV command somehow? Or maybe you have other ideas what is wrong?

 

Thanks in advance for your help,
Patryk 

0 Kudos
Highlighted
Voyager
Voyager
2,324 Views
Registered: ‎06-24-2013

Re: U-BOOT Networkboot - loading *.elf file for another core

Hey @oleniuk,

 

Could you upload the Linux boot log (dmesg) for both cases?

 

Thanks in advance,

Herbert

-------------- Yes, I do this for fun!
0 Kudos
Highlighted
Visitor
Visitor
2,314 Views
Registered: ‎08-29-2017

Re: U-BOOT Networkboot - loading *.elf file for another core

Hi @hpoetzl,

The dmesgs are attached, there are only few differences: 

 

The main is :

Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 11672K (df49a000 - e0000000)

It is not present in the "working FSBL" because in this case we're creating kernel+ramdisk+system.dtb together in uImage for Linux whereas when network booting Linux in u-boot the uramdisk.gz, system.dtb and image.ub are separate. 

 

There are also 2 other differences with addressing and available memory. 

 

Let me know if you could conclude something. 

 

Best,
Patryk 

0 Kudos
Highlighted
Visitor
Visitor
178 Views
Registered: ‎08-06-2019

Re: U-BOOT Networkboot - loading *.elf file for another core

For the 2nd core, you need to convert the elf file into bin file, like

arm-none-eabi-objcopy -O binary hello.elf hello.bin

tftpboot it right after you tftpboot the first core. like

tftpboot 0x0 cpu0.elf; tftpboot 0x3000000 cpu1.bin; bootelf 0x0;

CPU0 will start first and then you need to fire up CPU1 by calling some functions in CPU0 application. 

0 Kudos