UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Visitor jlfohr
Visitor
713 Views
Registered: ‎12-19-2017

Zynq AMP - Linux core affecting Baremetal core jitter

Jump to solution

I have a zedboard that I have setup to run AMP according to the xapp1078. Linux is running on CPU0 and is setup to use DDR from 0x0 to 0x17FF_FFFF. I have then created a baremetal application that runs on CPU1. This application has a single hardware interrupt generated from a AXI_timer peripheral that is hooked up to the private peripheral interrupt for CPU1, which is 31. It is triggered at a 20kHz rate. The entire application including bsp is small enough to fit in the OCM, so I have edited my linker file to place all of the sections in ps7_ram_0. I use ps7_ram_1 for data that I am manipulating in the interrupt. I also have a DDR section mapped starting at 0x1800_0000, which is currently not used.

Inside my interrupt I call XTime_GetTime() between the code execution time I want to measure. I then keep a max and min value of the difference and let the interrupt run for a long period of time and then print out the value to a terminal using the ps_uart1. What I notice is that if I don't do anything on my linux terminal I get a normal processing time of about 20us and a jitter of about 1us. However, if I run any command on the linux terminal, as simple as "ls -l" the max time seen jumps to about 24us and I get a jitter of 4us.

The linux system and FPGA image boots from an SDcard, and the application on CPU1 I am loading using the SDK. I configured it to not run the init scripts on launch, so I only reset CPU1 and load and run the application code. Since Linux remaps the OCM to high memory, I have also edited the default ram0 region to point to 0xFFFC_0000. I left the ram1 region alone which is pointing to 0xFFFF_0000.

The DDR is not used by CPU1. Xapp1078 seems to indicate that the L2 cache is not used on CPU1 if the define USE_AMP is set in the BSP, which I have done. The L1 cache is private between the CPUs. So, I'm confused as to what could be causing the extra jitter. Has anyone else encountered this or knows what is going on?

 

 

Tags (4)
0 Kudos
1 Solution

Accepted Solutions
Visitor jlfohr
Visitor
611 Views
Registered: ‎12-19-2017

Re: Zynq AMP - Linux core affecting Baremetal core jitter

Jump to solution

This is for SDK 2018.2. In the bsp generated for cpu1 there should be a file called boot.S

Inside there is a section of code that looks like this:

	/* Write to ACTLR */
	mrc	p15, 0, r0, c1, c0, 1		/* Read ACTLR*/
	orr	r0, r0, #(0x01 << 6)		/* set SMP bit */
	orr	r0, r0, #(0x01 )		/* Cache/TLB maintenance broadcast */
	mcr	p15, 0, r0, c1, c0, 1		/* Write ACTLR*/

Since we are running with the define USE_AMP set, this will disable the L2 cache. If you remove the set SMP bit instruction, so that you get the below code, it will prevent CPU1 from responding to the broadcast cache instructions.

	/* Write to ACTLR */
	mrc	p15, 0, r0, c1, c0, 1		/* Read ACTLR*/
#if USE_AMP != 1
	orr	r0, r0, #(0x01 << 6)		/* set SMP bit */
#endif
	orr	r0, r0, #(0x01 )		/* Cache/TLB maintenance broadcast */
	mcr	p15, 0, r0, c1, c0, 1		/* Write ACTLR*/

This is much cleaner than modifying the kernel code and should probably be the default when defining USE_AMP. 

 

****One caveat of not being in SMP mode. If you ever want to use the DDR on CPU1, you will have to mark the used regions as non-shareable otherwise it will treat the regions as non-cacheable, which will greatly reduce performance. In my case I'm using OCM to share data so the DDR regions are completely separate and have no reason to be shared.

0 Kudos
4 Replies
Visitor jlfohr
Visitor
660 Views
Registered: ‎12-19-2017

Re: Zynq AMP - Linux core affecting Baremetal core jitter

Jump to solution

Seems like it could be related to this post

https://forums.xilinx.com/t5/Embedded-Linux/High-bare-metal-timing-variablity-in-AMP-configuration-due-to/td-p/567124

Which seems to point to an issue with the Linux kernel code. 

0 Kudos
Visitor jlfohr
Visitor
636 Views
Registered: ‎12-19-2017

Re: Zynq AMP - Linux core affecting Baremetal core jitter

Jump to solution

So it appears this is the issue. If you comment out the ALT_SMP lines in the file "arch/arm/mm/cache-v7.s" in the linux kernel and replace the ALT_UP with ALT_SMP, the extra jitter goes away. This seems to imply that if you compile the kernel as SMP, that even if you set maxcpus=1 in the bootargs that it will still cause some interaction on the CPU1 side.

0 Kudos
Visitor jlfohr
Visitor
627 Views
Registered: ‎12-19-2017

Re: Zynq AMP - Linux core affecting Baremetal core jitter

Jump to solution
/*
 *	v7_flush_icache_all()
 *
 *	Flush the whole I-cache.
 *
 *	Registers:
 *	r0 - set to 0
 */
ENTRY(v7_flush_icache_all)
	mov	r0, #0
	ALT_SMP(mcr	p15, 0, r0, c7, c1, 0)		@ invalidate I-cache inner shareable
	ALT_UP(mcr	p15, 0, r0, c7, c5, 0)		@ I+BTB cache invalidate
	ret	lr
ENDPROC(v7_flush_icache_all)

The ALT_SMP code is an ICIALLUIS instruction which means Instruction Cache Invalidate All to PoU, Inner Shareable. 

The ALT_UP code is an ICIALLU instruction which means Instruction Cache Invalidate All to PoU

According to this link https://community.arm.com/processors/f/discussions/5958/armv7-iciallu-vs-icialluis

The ICIALLU only affects the one processor and invalidates to the Point of Unification. On the other hand ICIALLUIS is like a broadcast and essentially tells every core in the same inner shareable domain to invalidate to their PoU.

Therefore, as the default kernel is built using the SMP option, this will cause both CPU's to invalidate their L1 caches when called regardless of the maxcpu directive.

So options I see are you either build the kernel as uniprocessor, figure out how to remove CPU1 from the inner shareable domain of CPU0, or manually patch the kernel code to prevent these broadcast instructions from happening.

0 Kudos
Visitor jlfohr
Visitor
612 Views
Registered: ‎12-19-2017

Re: Zynq AMP - Linux core affecting Baremetal core jitter

Jump to solution

This is for SDK 2018.2. In the bsp generated for cpu1 there should be a file called boot.S

Inside there is a section of code that looks like this:

	/* Write to ACTLR */
	mrc	p15, 0, r0, c1, c0, 1		/* Read ACTLR*/
	orr	r0, r0, #(0x01 << 6)		/* set SMP bit */
	orr	r0, r0, #(0x01 )		/* Cache/TLB maintenance broadcast */
	mcr	p15, 0, r0, c1, c0, 1		/* Write ACTLR*/

Since we are running with the define USE_AMP set, this will disable the L2 cache. If you remove the set SMP bit instruction, so that you get the below code, it will prevent CPU1 from responding to the broadcast cache instructions.

	/* Write to ACTLR */
	mrc	p15, 0, r0, c1, c0, 1		/* Read ACTLR*/
#if USE_AMP != 1
	orr	r0, r0, #(0x01 << 6)		/* set SMP bit */
#endif
	orr	r0, r0, #(0x01 )		/* Cache/TLB maintenance broadcast */
	mcr	p15, 0, r0, c1, c0, 1		/* Write ACTLR*/

This is much cleaner than modifying the kernel code and should probably be the default when defining USE_AMP. 

 

****One caveat of not being in SMP mode. If you ever want to use the DDR on CPU1, you will have to mark the used regions as non-shareable otherwise it will treat the regions as non-cacheable, which will greatly reduce performance. In my case I'm using OCM to share data so the DDR regions are completely separate and have no reason to be shared.

0 Kudos