cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted
Adventurer
Adventurer
12,855 Views
Registered: ‎09-28-2016

how disable cache in Linux

Jump to solution
hi,all

I have a zc706 board that run petalinux2015.4, now I want to disable the cache in Linux application, how can I do it?

thanks a lot


Tom
0 Kudos
1 Solution

Accepted Solutions
Highlighted
Scholar
Scholar
17,690 Views
Registered: ‎05-28-2013

Tom> you mean that the cache default enabled in linux kernel after linux system startup?

Yes. During boot up (before Linux) the caches are enabled and disabled, possibly several times. Once Linux is running, the caches are normally enabled. If you want to change that, you can change the kernel options (eg. rebuild the kernel).

Tom> is there any API function get the cache state in linux user space or bare metal application? enabled or disabled!!!

This is very system-specific and therefore there is not standard API for accessing it. However, on the ARM you can read it from the System Control Register, using assembly code like the following:

MRC p15, 0,<Rd>, c1, c0, 0; Read SCTLR

You will be interested in bit[2] and bit[12] which tell you about the state of the data and instruction caches, respectively. For details please see http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.100511_0401_10_en/ric1447333674419.html

 

Note that if you have caches disabled, Linux boot time will be significantly longer. It may run 2x or 10x slower than normal. The difference will be very obvious, so maybe you do not need any code to check it.

View solution in original post

26 Replies
Highlighted
Moderator
Moderator
12,775 Views
Registered: ‎12-04-2016

Hi Tom

 

In your Application code, you can include "xil_cache.h" file, which has a set of API's related to Various Cache Functions.

 

 

To Disable the caches, you can make use of the following calls in your Application:

 

Xil_ICacheDisable();

Xil_L1ICacheDisable();

Xil_L2CacheDisable();

 

Please Refer the Below link, to know more about this:

https://www.xilinx.com/support/documentation/sw_manuals/xilinx2014_1/oslib_rm.pdf 

 

 

Best Regards

Shabbir

 

 

 

 

 

0 Kudos
Highlighted
Adventurer
Adventurer
12,753 Views
Registered: ‎09-28-2016

Hi,shabbirk

 

thanks for your response.

 

Xil_ICacheDisable();

Xil_L1ICacheDisable();

Xil_L2CacheDisable();

 

I think these three functions only work in standalone application, if I want to handle cache in linux application. how should I do?

 

Tom

0 Kudos
Highlighted
Moderator
Moderator
12,736 Views
Registered: ‎12-04-2016

Hi Tom

 

Yes I can Understand your Requirement for Disabling cache at Linux User Space. But as far as I know, cache disabling can only be done by first switching from user mode to SVC mode and then you can use asm volatile C equivalent Instructions to perform Cache disabling.

 

May be the below link gives you some insight into this:

https://www.community.arm.com/processors/b/blog/posts/caches-and-self-modifying-code

 

 

Thanks & Best Regards

Shabbir

0 Kudos
Highlighted
Scholar
Scholar
12,721 Views
Registered: ‎05-28-2013

Hi Tom,

 

Caches are normally enabled by the bootloader, and then disabled again just before starting the Linux kernel, where they get enabled again. So to switch them off, you might need to modify the bootloader, and you definitely need to modify the Linux kernel configuration.

 

Kernel configuration options you may need to toggle include: CONFIG_CPU_ICACHE_DISABLE and CONFIG_CPU_DCACHE_DISABLE, and possibly CONFIG_CACHE_L2X0. Keep in mind that this will have quite noticeable performance impact, your system will boot much more slowly.

 

 

0 Kudos
Highlighted
Adventurer
Adventurer
12,704 Views
Registered: ‎09-28-2016

Hi,

 

thanks for your clarification.

 

you mean that the cache default enabled in linux kernel after linux system startup?

 

is there any API function get the cache state in linux user space or bare metal application? enabled or disabled!!!

 

 

Tom

0 Kudos
Highlighted
Scholar
Scholar
17,691 Views
Registered: ‎05-28-2013

Tom> you mean that the cache default enabled in linux kernel after linux system startup?

Yes. During boot up (before Linux) the caches are enabled and disabled, possibly several times. Once Linux is running, the caches are normally enabled. If you want to change that, you can change the kernel options (eg. rebuild the kernel).

Tom> is there any API function get the cache state in linux user space or bare metal application? enabled or disabled!!!

This is very system-specific and therefore there is not standard API for accessing it. However, on the ARM you can read it from the System Control Register, using assembly code like the following:

MRC p15, 0,<Rd>, c1, c0, 0; Read SCTLR

You will be interested in bit[2] and bit[12] which tell you about the state of the data and instruction caches, respectively. For details please see http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.100511_0401_10_en/ric1447333674419.html

 

Note that if you have caches disabled, Linux boot time will be significantly longer. It may run 2x or 10x slower than normal. The difference will be very obvious, so maybe you do not need any code to check it.

View solution in original post

Highlighted
Adventurer
Adventurer
12,661 Views
Registered: ‎09-28-2016

Hi,

 

thanks a lot for your clarification. I got it

 

 

Tom

0 Kudos
Highlighted
Explorer
Explorer
12,637 Views
Registered: ‎06-03-2015

Hi Tom,

 

In continuation of above discussion, Icache and Dcache are part of ARM Linux Boot Sequence,There are two stages in Booting Linux system one is Platfrom dependent Initilzation & Platform Independent Initilization.

In First part, Icache is definitely off and Dcache can be off or on. In Second part (Platform Independent Intilization), Icache and Dcache should be enable through Linux kernel using "start_kernel()" function.

 

Icache and Dcache are not supposed to be touched later on, as these are Initial Functions to make corrspending I/O peripherals work properly. So enabling/disabling of these functions are not good Idea.

 

Thanks & Regards
Satish.G

 

 

 

 

 

 

 

 

 

 

 

 

G Satish Kumar
0 Kudos
Highlighted
Explorer
Explorer
12,635 Views
Registered: ‎06-03-2015

Hi Tom,

 

In continuation of above discussion, Icache and Dcache are part of ARM Linux Boot Sequence,There are two stages in Booting Linux system one is Platfrom dependent Initilzation & Platform Independent Initilization.

In First part, Icache is definitely off and Dcache can be off or on. In Second part (Platform Independent Intilization), Icache and Dcache should be enable through Linux kernel using "start_kernel()" function.

 

Icache and Dcache are not supposed to be touched later on, as these are Initial Functions to make corrspending I/O peripherals work properly. So enabling/disabling of these functions are not good Idea.

 

Thanks & Regards
Satish.G 

G Satish Kumar
0 Kudos
Highlighted
Adventurer
Adventurer
9,793 Views
Registered: ‎09-28-2016

HI, Satish

 

Thanks for your clarification.

 

I want to consult you another two questions, please.

1.did the Dcache that you mentioned include L2 cache? or just only L1 Dcache?

2.how can I set one range of memory that reserved to uncache property through mmap function or other ways?

 

 

Tom

 

 

 

0 Kudos
Highlighted
Explorer
Explorer
9,779 Views
Registered: ‎06-03-2015

Hi Tom,

 

1.did the Dcache that you mentioned include L2 cache? or just only L1 Dcache?

 

[Satish]: In ARM Arch.. each CPU Core contains its own L1 cache (Icache & Dcache) and single L2 cache whicih is common to ALL CPU Cores.Example if Zc702 is having two ARM Cores , there are Two L1 caches and One L2 cache.

 

In the time of Booting, only Boot CPU Core (Probably CPU0) will enable the corrpsending L1Cache (Icache & Dcache) for Fast Excueation. Remaining Cores are get enabled through start_kernel () function where it calls other Cores corrspending L1 Cache  which will be enabled through SMP (Symmetric Multi Processing )functions.

 

L2 cache is taking care of simulation among/between these L1 caches. 

So Dcache mentioned in the earlier discussion is regards to L1 cache of Boot CPU Core. 

Where L1 cache is combination of Icache and Dcache.

 

 

2.how can I set one range of memory that reserved to uncache property through mmap function or other ways?

[Satish]:  I am not understanding your question properly, but based on initial Discussions, If CPU0 and CPU1 wants to share the memory , we use the shared memory region, It declares set of Memory areas for CPU0 and CPU1 when it wants to share the memory area , it will be declared in that area,(Part of DDR) instead of using mmap.

mmap is used to map device memory from user space specially when writng ioctl functionalities and when devolope user space device drivers.

 

But Internally to share the data between different CPUs shared memory, L2 cache, L1 Caches are internally taking care of the same.

 

Thanks & Regards
Satish.G
 

 

 

 

G Satish Kumar
0 Kudos
Highlighted
Adventurer
Adventurer
4,238 Views
Registered: ‎10-02-2018

Hi @gsatish10 , @atower , @rfs613 , @shabbirk 

Have you try disable cache in petalinux successfull?

I 'm use zc706 and need to disable cache to run cdma.

Thanks

0 Kudos
Highlighted
Scholar
Scholar
4,178 Views
Registered: ‎05-28-2013

@sonminh wrote:

 

Have you try disable cache in petalinux successfull?

I 'm use zc706 and need to disable cache to run cdma.


Under Linux, you can disable the cache using the following kernel configuration options:CONFIG_CPU_ICACHE_DISABLE and CONFIG_CPU_DCACHE_DISABLE, and possibly CONFIG_CACHE_L2X0. Note you will need to rebuild the kernel to change these settings.

 

However the effect is global and will significanly slow down everything, from boot onwards. So except for testing purposes, you really don't want to disble the caches. A better approach is to fix your driver...

0 Kudos
Highlighted
Adventurer
Adventurer
4,153 Views
Registered: ‎10-02-2018

Hi @rfs613 ,

I just try implement as you talk but no have any change once boot petalinux 2017.4 on zc706:

1. In file xilinx_zynq_defconfig of external linux  (components/ext_sources/linux-xlnx-2017.4_video_ea/arch/arm/configs), I add in to end of file:

CONFIG_CPU_ICACHE_DISABLE=y
CONFIG_CPU_DCACHE_DISABLE=y

2. petalinux-config -c kernel --defconfig xilinx_zynq_defconfig 

3. petalinux-build -c kernel

4. Modify file system-user.dtsi in /project-spec/meta-user/recipes-bsp/device-tree/files:

/include/ "system-conf.dtsi"
/ {
};
&amba {
cache-controller@f8f02000 {
status = "disabled";
};
};

5. Petalinux-build

This is my device-tree after disable cache and build:

cache-controller@f8f02000 {
compatible = "arm,pl310-cache";
reg = <0xf8f02000 0x1000>;
interrupts = <0x0 0x2 0x4>;
arm,data-latency = <0x3 0x2 0x2>;
arm,tag-latency = <0x2 0x2 0x2>;
cache-unified;
cache-level = <0x2>;
status = "disabled";
};

But once boot with BOOT.bin and image.ub just build, L2 cache still enable and I don't know veryfy that L1 is disabled. I see time to boot still fast. Here is console once boot linux, L2 cache still probe although disable.

Starting kernel ...

Uncompressing Linux... done, booting the kernel.
Booting Linux on physical CPU 0x0
Linux version 4.9.0-xilinx (thanhnt@ubuntu) (gcc version 6.2.1 20161016 (Linaro GCC 6.2-2016.11) ) #1 SMP PREEMPT Sat Nov 16 14:11:29 +07 2019
CPU: ARMv7 Processor [413fc090] revision 0 (ARMv7), cr=18c5287d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
OF: fdt:Machine model: Zynq ZC706 Development Board
bootconsole [earlycon0] enabled
cma: Reserved 16 MiB at 0x3f000000
Memory policy: Data cache writealloc
percpu: Embedded 14 pages/cpu @ef7cb000 s25932 r8192 d23220 u57344
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 260608
Kernel command line: console=ttyPS0,115200 earlyprintk
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 1007240K/1048576K available (6144K kernel code, 200K rwdata, 1464K rodata, 1024K init, 230K bss, 24952K reserved, 16384K cma-reserved, 245760K highmem)
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
    vmalloc : 0xf0800000 - 0xff800000   ( 240 MB)
    lowmem  : 0xc0000000 - 0xf0000000   ( 768 MB)
    pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
    modules : 0xbf000000 - 0xbfe00000   (  14 MB)
      .text : 0xc0008000 - 0xc0700000   (7136 kB)
      .init : 0xc0900000 - 0xc0a00000   (1024 kB)
      .data : 0xc0a00000 - 0xc0a32180   ( 201 kB)
       .bss : 0xc0a32180 - 0xc0a6b998   ( 231 kB)
Preemptible hierarchical RCU implementation.
        Build-time adjustment of leaf fanout to 32.
        RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
RCU: Adjusting geometry for rcu_fanout_leaf=32, nr_cpu_ids=2
NR_IRQS:16 nr_irqs:16 16
efuse mapped to f0802000
slcr mapped to f0804000
L2C: platform modifies aux control register: 0x72360000 -> 0x72760000
L2C: DT/platform modifies aux control register: 0x72360000 -> 0x72760000
L2C-310 enabling early BRESP for Cortex-A9
L2C-310 full line of zeros enabled for Cortex-A9
L2C-310 ID prefetch enabled, offset 1 lines
L2C-310 dynamic clock gating enabled, standby mode enabled
L2C-310 cache controller enabled, 8 ways, 512 kB
L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x76760001
zynq_clock_init: clkc starts at f0804100
Zynq clock init

Can you talk clearly? Thank you

 

0 Kudos
Highlighted
Scholar
Scholar
4,099 Views
Registered: ‎05-28-2013

I would suggest to confirm that the kernel and DTS changes actually got included in your build files:

  • zgrep CACHE /proc/config.gz  --- you can check your kernel CONFIG settings here
  • Somewhere in /proc/device-tree/... you should be able to find the DTS section for cache-controller@f8f02000. It will be a directory, containing a file called "status".  You can "cat" the status file to check that it is disabled.

If these no not match, then something in your build/boot process is not right, eg. your changes are not making it through.

 

For the outer pl310 cache, there are some additional kernel config settings. On my system for example, I have CONFIG_CACHE_L2X0=y (but this a different CPU than you are using).

 

Please check earlier message in this thread about how to check if the cache is disabled. But really, the difference is very visible on the serial console, booting takes considerably longer.

 

Let me re-emphasise that switching off caches under Linux is very uncommon. It is not a configuration that gets much testing (if any), and you won't find much information about it.

0 Kudos
Highlighted
Adventurer
Adventurer
4,036 Views
Registered: ‎10-02-2018

Hi @rfs613 

How can enable zgrep command in petalinux.

I just try zgrep but console notify:

-sh: zgrep: command not found

0 Kudos
Highlighted
Scholar
Scholar
4,020 Views
Registered: ‎05-28-2013

@sonminh wrote:

Hi @rfs613 

How can enable zgrep command in petalinux.

Instead of zgrep, you can use:

gzip -dc /proc/config.gz | grep CACHE

(To answer the question, zgrep command comes from the gzip package)

 

0 Kudos
Highlighted
Visitor
Visitor
3,613 Views
Registered: ‎12-15-2019

I have done it on armv8-a linux. I do it not to measure performance but to verify that xilinx zcu104 platform might have potential coherence error. As a result, the pynq image xilinx provided must have some coherence error during pl and ps communication. Here is my workaround:

  1. My platform is cortex-a53, ubuntu18 started at EL2 and switched to EL1 and supported SMP on four cpu cores. Thus, I need to turn off multi-core to ensure L2-cache coherency. Thanks to the feature of cpu-hot-plug, I just run :

    echo '0' > /sys/devices/system/cpu/cpu1/online,

    echo '0' > /sys/devices/system/cpu/cpu2/online,

    echo '0' > /sys/devices/system/cpu/cpu3/online

    then I run dmesg to verify that multi-core has been turn off.

  2. I build the kernel source tree, cause I cannot find it in my linux. You can run uname -r to see your kernel version. And find in /usr/src to see whether your linux already have it.

  3. I build the linux module. With gcc inline asm , I flush all cache and set sctlr_el1.c 0.

  4. I insmod the module. And I first run my program with the right result, although it takes 20 times slower than multi-core and D-cache on.

0 Kudos
Highlighted
Visitor
Visitor
3,472 Views
Registered: ‎01-31-2019

hi hht,

Could you tell me the details in step 3?  It will be helpful if you provide the source of module.  I want to disable cache on my zcu104 platform. 

Thanks.

0 Kudos
Highlighted
Visitor
Visitor
3,441 Views
Registered: ‎12-15-2019
https://github.com/i24361/DCacheDisable,you can directly use my module, if your linux version is xilinx 2018.3.
0 Kudos
Highlighted
Adventurer
Adventurer
527 Views
Registered: ‎07-24-2020

Hi @rfs613 

I want to boot Linux(Ubuntu) on the Zynq board by disabling both L1 and L2 caches, which means when OS is up and running, cache has to be disabled. For that, I have disabled the cache in Zynq 7000 by following below steps, I have noticed that boot time is now 10 times slower than a normal boot. Is there any other way that I can verify both CPU cache(L1) and L2 cache are disabled?

Steps followed:

Added Below parameters on arch/arm/configs/xilinx_zynq_defconfig

cat <<! >> arch/arm/configs/xilinx_zynq_defconfig
CONFIG_RELOCATABLE=y
CONFIG_CPU_ICACHE_DISABLE=y
CONFIG_CPU_DCACHE_DISABLE=y
!

Removed L2 cache from the device tree by commenting out the cache controller section from zynq-7000.dtsi device tree source file, and Rebuilt the device tree blob:

dtc -I dts -O dtb -o devicetree.dtb system.dts

I verified /proc/config.gz and found only CONFIG_CPU_ICACHE_DISABLE=y.

 

ubuntu@arm:~$ gzip -dc /proc/config.gz | grep CACHE
CONFIG_CPU_CACHE_V7=y
CONFIG_CPU_CACHE_VIPT=y
CONFIG_CPU_ICACHE_DISABLE=y
# CONFIG_CPU_ICACHE_MISMATCH_WORKAROUND is not set
CONFIG_OUTER_CACHE=y
CONFIG_OUTER_CACHE_SYNC=y
CONFIG_MIGHT_HAVE_CACHE_L2X0=y
CONFIG_CACHE_L2X0=y
# CONFIG_CACHE_L2X0_PMU is not set
CONFIG_ARM_L1_CACHE_SHIFT_6=y
CONFIG_ARM_L1_CACHE_SHIFT=6
# CONFIG_CLEANCACHE is not set
CONFIG_DST_CACHE=y
CONFIG_DCACHE_WORD_ACCESS=y
CONFIG_FS_MBCACHE=y
# CONFIG_FSCACHE is not set

 

But I couldn't find 'CONFIG_CPU_DCACHE_DISABLE' and 'CONFIG_RELOCATABLE'. What does this mean? Is there any way to confirm that both caches are disabled?

Regards,

Prasanna

0 Kudos
Highlighted
Scholar
Scholar
499 Views
Registered: ‎05-28-2013

Hi @Prasanna_K 

I suspect some of the kernel options have been renamed and/or split up into multiple options. You are probably using different kernel version (newer or older). It looks like at least CONFIG_CPU_ICACHE_DISABLE is still the same. You'll have to have a look in the kernel settings (in the kernel source tree, run "make menuconfig", then you can search for "cache"). Sorry I can't be more specific.

I'm not sure where CONFIG_RELOCATABLE came from. This may well be a valid setting, but it has nothing to do with the caches.

Besides the greatly reduced boot speed, and the lack of messages about pl310 in the boot messages, the only other way you can check that caches are disabled, is to read from the system control registers. I explained how to do this using an "MRC" instruction in an earlier reply in this thread (the one marked as "solution").

0 Kudos
Highlighted
Adventurer
Adventurer
496 Views
Registered: ‎07-24-2020

Hi @rfs613 ,

Thanks for your response, I got the kernel source from Xilinx github, I shall take a look at menuconfig to see more options about caches.

This is very system-specific and therefore there is not standard API for accessing it. However, on the ARM you can read it from the System Control Register, using assembly code like the following:

MRC p15, 0,<Rd>, c1, c0, 0; Read SCTLR

 


Is there a document or article explaining how to see the system control register? I have gone through the link and But not sure how can I view these values in Zynq. It would be great if you could share the procedure to view this register.

Regards,

Prasanna

0 Kudos
Highlighted
Scholar
Scholar
458 Views
Registered: ‎05-28-2013


Is there a document or article explaining how to see the system control register? I have gone through the link and But not sure how can I view these values in Zynq. It would be great if you could share the procedure to view this register.

I googled and found this: https://community.arm.com/developer/ip-products/processors/f/cortex-a-forum/11148/how-to-access-the-system-control-register specifically the 32-bit sample code. This shows how you can perform the "MRC" assembly instruction, and how to print out the resulting value.

There is one potential issue though. The system control register can only be accessed in "privileged mode" (which basically means kernel level). So, I don't think you can call it from main() as shown. Instead, you likely will need to modify the linux kernel source code, to do the MRC instruction during the kernel boot. Then you can print the result using the kernel printk() function.

0 Kudos
Highlighted
Adventurer
Adventurer
373 Views
Registered: ‎07-24-2020

Hi @rfs613 ,

Thanks for sharing the link,

I have created the below file  and tried to compile using 'gcc -c sctrl.c -o sctrl.o' but I am getting an error saying no such instruction.

 

#include <stdio.h>

unsigned int A_bit()
{
	  unsigned int res = 0;

	  __asm volatile
	   (
		"MRC p15, 0, %[result], c1, c0, 0": [result] "=r" (res)
           );

	   return res;
}
int main(void)
{
	  unsigned int sys_reg = 0;

	  sys_reg = A_bit();
	 
	  printf("system register value is %d \n", sys_reg);
	  
}

 

 

Error:

 

[kotrappa@loke drivers]$ gcc -c sctrl.c -o sctrl.o
sctrl.c: Assembler messages:
sctrl.c:7: Error: no such instruction: `mrc p15,0,%eax,c1,c0,0'

 

 

Any idea what is the issue here?

Regards,

Prasanna

0 Kudos
Highlighted
Scholar
Scholar
364 Views
Registered: ‎05-28-2013

[kotrappa@loke drivers]$ gcc -c sctrl.c -o sctrl.o
sctrl.c: Assembler messages:
sctrl.c:7: Error: no such instruction: `mrc p15,0,%eax,c1,c0,0'

Any idea what is the issue here?

It looks like you are compiling this on your PC, which has an x86 processor. This will not work, because "mrc" instruction only exists on the ARM processor. The x86 compiler does not know what "mrc" is, hence the error message.

You need to use a cross compiler (arm-linux-*-gcc) instead of the host (x86) gcc. You may also need to specify the ARM processor type (Cortex-A9).

0 Kudos