UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Observer masrut
Observer
10,028 Views
Registered: ‎07-30-2015

USB and L2 cache in AMP system

Jump to solution

I have an Zynq AMP system, running Linux 2015.2.1 on CPU0 and Bare metal on CPU1, which is a little bit different from XAPP1078, because I have L2 cache controller disabled in Linux and L2 Cache enabled in Bare Metal. I need speed in the bare metal app, but I need Linux just for the USB peripherals.

Now the system seems to work, but when the Bare Metal is running, if I insert a USB memory dongle the enumeration fails. I debugged the problem and it seems that it starts when I enable the L2 cache controller in BOOT.S file. It seems that the USB peripheral accesses the L2 cache bypassing the L2 cache controller (I disabled the L2 cache controller in the device tree), so when I enable cache for the CPU1 there is some conflict. 

Does anybody know how to disable L2 cache completely for the USB in Zynq ?

 

Max

Tags (1)
0 Kudos
1 Solution

Accepted Solutions
Observer masrut
Observer
17,014 Views
Registered: ‎07-30-2015

Re: USB and L2 cache in AMP system

Jump to solution

I decided to disable L2 cache in the bare metal app as it is in the official solution, and now my system works.

I see that the increment in speed because of L2 cache is not what I thought...

0 Kudos
11 Replies
Highlighted
Xilinx Employee
Xilinx Employee
10,023 Views
Registered: ‎09-10-2008

Re: USB and L2 cache in AMP system

Jump to solution

Hi Max,

 

I don't know the answer but have some other thoughts.  Are you sure you can really do that as there is no disable of L2 for Linux while enable it for the other CPU in my thinking.  There is a single enable and if Linux sees it enabled my guess is that it's going to use it also.  I don't believe that any device accesses the L2 specifically they just access memory that the L2 is caching. If you leave the L2 off on both CPUs I would think it would work but be slow?

 

Thanks

John

 

0 Kudos
Observer masrut
Observer
10,017 Views
Registered: ‎07-30-2015

Re: USB and L2 cache in AMP system

Jump to solution

Well, I think that Linux does not L2 cache enabled because I commented the L2 controller in the device tree.

Then, I modified BOOT.S in order to enable cache from address to 0x1000000 to 0x1ffffff for CPU1, and set ram from 0x00000000 to 0x0fffffff as unavailable for CPU1. Linux has been made aware of RAM only from 0x00000000 to 0x0ffffffff, by modifiyng the devicetree, so CPU0 can only see shared resources with CPU1 and not DDR RAM. 

 

From what I see, it seems that the USB interface driven by LINUX accesses the L2 cache and not the DDR directly, because when I enable the L2 cache in CPU1 by the following code I see USB on CPU0 beginning doing errors.

 

CODE RUNNING ON CPU1 (BOOT.S)  :

ldr r0,=L2CCCrtl         /* Load L2CC base address base + control register */
ldr r1,[r0]              /* read the register */
mov r2, #L2CCControl     /* set the enable bit */
orr r1,r1,r2
str r1, [r0]             /* enable the L2 Caches */ ====> with this instruction, USB on CPU0 fails

 

So.. I am stuck... I beggining to think that the USB interface has its own L2 cache controller wich is uncontrollable by the devicetree...

 

Max

 

 

 

0 Kudos
Xilinx Employee
Xilinx Employee
9,994 Views
Registered: ‎09-10-2008

Re: USB and L2 cache in AMP system

Jump to solution

Hi Max,

 

Again, just brainstormning here, but I'm 99% sure your headed down the wrong path with the USB controller having it's own L2 cache controller.  I would think it's more likely that one of your changes is not doing what you think with respect to the L2 cache controller. Since you don't see the issue til CPU1 enables the L2 I would think maybe your changes to CPU1 BSP are not complete or correct such that it is maybe still setting up some memory that CPU0 uses? 

 

I would double check all your changes in other ways besides just the code (in runtime from a Linux console, amount of memory and L2 cache not enabled in device tree, etc...) when possible.  What about invalidating the L2 cache before you enable it in CPU1 also to ensure there's nothing in there that you don't want?  I would not think this is necessary as FSBL does that before handing off to u-boot, but maybe u-boot is using the L2 such that there is stuff in it.

 

Thanks

John

 

 

0 Kudos
Observer masrut
Observer
9,983 Views
Registered: ‎07-30-2015

Re: USB and L2 cache in AMP system

Jump to solution

John, 

 

this is just brainstorming but I thank you so much for your help because it helps a lot, and talking with an expert contributor as you are for me is very helpful.

Anyway, my AMP system with bare metal on CPU1 with L2 enables and linux on CPU0 with L2 disabled has worked well for a long time, until I decided to add USB support to Linux. So it is impossible that the L2 cache is dirty because of the FSBL, otherwise my system would not have worked without USB support.

Everyting began because of USB.

Today I figured out what is going on, ... maybe.,... because I am not expert....

The USB driver of Linux is based on DMA accesses to the RAM. The driver allocates memory space for the USB buffers.

I think that the USB driver also writes to CPU0 MMU in order to declare L2 cached the buffers, even if the L2 controller has been removed from the devicetree. When I enable the L2 cache controller by CPU1, the USB DMA accesses to memory use the L2 controller just because I enabled it by CPU1 even if in the devicetree the L2 controller is disabled. At least, I suppose.  This could be a bug of the USB driver... maybe...

Please, can you give your opinion on this ?

Thanks in advance, 

Max

 

 

 

 

 

 

 

 

 

0 Kudos
Xilinx Employee
Xilinx Employee
9,979 Views
Registered: ‎09-10-2008

Re: USB and L2 cache in AMP system

Jump to solution

Hi Max,

 

The bug in the driver sounds unlikely to me as drivers don't usually do anything with cache controllers. Yes they do allocate memory and alter MMU settings for that memory.  Maybe you've always had a memory setup bug that you didn't realize because Linux was never touching that memory til USB was used?

 

Thanks

John

 

0 Kudos
Observer masrut
Observer
9,977 Views
Registered: ‎07-30-2015

Re: USB and L2 cache in AMP system

Jump to solution

Yes... this is possible... but how to discover where it is  ??

0 Kudos
Xilinx Employee
Xilinx Employee
9,975 Views
Registered: ‎09-10-2008

Re: USB and L2 cache in AMP system

Jump to solution
I would thoroughly review the boot.S changes you made for CPU1 and verify anything from Linux runtime that might verify it's memory also (/proc/device-tree/, etc...).
0 Kudos
Observer masrut
Observer
9,959 Views
Registered: ‎07-30-2015

Re: USB and L2 cache in AMP system

Jump to solution

For the modifications in boot.s I think having done correctly...

I set from 0 to 1ff 0x15de6 which means cacheable for CPU1

Then I set from 0 to 0ff to 0x0 which means non accesible by CPU1

From Linux side I just removed the L2 controller from the devicetree, but maybe I also have to do

some other configuration in the kernel...

 

 

#if USE_LINUX==1
ldr r3, =0x1ff /* 256 entries to cover 256MB DDR */
ldr r0, =TblBase /* MMU Table address in memory */
ldr r2, = 0x15de6 /* S=b1 TEX=b101 AP=b11, Domain=b1111, C=b0, B=b1 */
mmu_loop1:
str r2, [r0] /* write the entry to MMU table */
add r0, r0, #0x4 /* next entry in the table */
add r2, r2, #0x100000 /* next section */
subs r3, r3, #1
bge mmu_loop1 /* loop till 256 MB is covered */
/********************************************************************/
/* Se LINUX la memoria da 0x00000000 a 0x0fffffff risulta riservata */
/********************************************************************/
ldr r3, =0x0ff /* 256 entries to cover 256MB DDR */
ldr r0, =TblBase /* MMU Table address in memory */
ldr r2, = 0x0000
mmu_loop:
str r2, [r0] /* write the entry to MMU table */
add r0, r0, #0x4 /* next entry in the table */
add r2, r2, #0x100000 /* next section */
subs r3, r3, #1
bge mmu_loop /* loop till 512MB is covered */
#endif
/*************************************************************************/

 

 

0 Kudos
Adventurer
Adventurer
9,945 Views
Registered: ‎11-05-2014

Re: USB and L2 cache in AMP system

Jump to solution

This code doesn't look right.  Your second loop loads r0 with TblBase again and then goes on to overwrite entries written by the first loop.  And same with r2.  By setting r2 to 0 in the second loop, the two loops end up setting MMU entries for the same addresses (The upper X bits of an MMU entry contains the 1MB address for which the MMU settings applies to).

 

I think you want something like this

 

#if USE_LINUX==1
/* Mark 0-256MB as unavailable to CPU1 */ ldr r3, =0xff /* 256 entries to cover 256MB DDR */ ldr r0, =TblBase /* MMU Table address in memory */ ldr r2, =0x0000 mmu_loop1: str r2, [r0] /* write the entry to MMU table */ add r0, r0, #0x4 /* next entry in the table */ add r2, r2, #0x100000 /* next section */ subs r3, r3, #1 bge mmu_loop1 /* loop till 256 MB is covered */
/* Mark 256MB-512MB as cacheable to CPU1 */ ldr r3, =0x0ff /* 256 entries to cover 256MB DDR */
ldr r2, =0x10015de6 /* Set MMU entry address to 0x100XXXXX (256MB) */
/* Do not reload r0, continue filling MMU table from where it was left off */

mmu_loop: str r2, [r0] /* write the entry to MMU table */ add r0, r0, #0x4 /* next entry in the table */ add r2, r2, #0x100000 /* next section */ subs r3, r3, #1 bge mmu_loop /* loop till 512MB is covered */ #endif

 

Of course, double check to make sure it's really what you need.  And no guarantees on whether I used the right assembly instruction/syntax, so check those too.

 

 

 

 

0 Kudos
Observer masrut
Observer
7,619 Views
Registered: ‎07-30-2015

Re: USB and L2 cache in AMP system

Jump to solution

Thank you for the hint, I tested it. Unfortunately nothing changes...

When I enable USB from CPU1 by the command:  *(unsigned int*)0xe0002170 = 0x600a0067;

(the write is needed because a known work around of Petalinux 2015.2.1)

 

I get this on the Lunix console from CPU0 :

 

usb 1-1: device descriptor read/64, error -32
usb 1-1: device descriptor read/64, error -32
usb 1-1: new high-speed USB device number 3 using ci_hdrc
usb 1-1: device descriptor read/64, error -32
usb 1-1: device descriptor read/64, error -32
usb 1-1: new high-speed USB device number 4 using ci_hdrc
usb 1-1: device not accepting address 4, error -32
usb 1-1: new high-speed USB device number 5 using ci_hdrc
usb 1-1: device not accepting address 5, error -32
usb usb1-port1: unable to enumerate USB device

 

Without USB enabled, everything works perfectly both on CPU0 and CPU1..

 

Max

 

0 Kudos
Observer masrut
Observer
17,015 Views
Registered: ‎07-30-2015

Re: USB and L2 cache in AMP system

Jump to solution

I decided to disable L2 cache in the bare metal app as it is in the official solution, and now my system works.

I see that the increment in speed because of L2 cache is not what I thought...

0 Kudos