Zynq MPSoC HPC0 Hadware Cache Coherency Not working

[fahadak69]

I am attempting AXI CDMA linux userspace design following instructions at the link below on my ZCU104 board

https://forums.xilinx.com/t5/Design-and-Debug-Techniques-Blog/AXI-CDMA-Linux-user-space-example-on-Zynq-UltraScale-RFSoC/ba-p/1096735#feedback-success

I need hardware coherency in my design (using HPC0 port) which is where I am having issues. 

I followed the link below to enable coherency in linux 

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842098/Zynq+UltraScale+MPSoC+Cache+Coherency

I have created an application for my board using example application provided in the link above. When I run the application and do a read back after cdma writing,  there is data mismatch between source (PS DDR) data and destination (FPGA BRAM) data. Many of the values readback from destination are unchanged 

To confirm that non-coherency is indeed causing problems here, I added O_SYNC attribute in the /dev/mem (i.e. instead of fd = open("/dev/mem", O_RDWR), I replaced it with fd = open("/dev/mem", O_RDWR | O_SYNC) in the example application) and now when I run application (i.e. write and read BRAM), all values match. Note that O_SYNC option basically asks for non-cache access to the memory. Using O_SYNC switch, all values match which confirm coherency as the problem here

I have verified all the required settings in Vivado as well as device register (enable CCI on HPC0, Enable snooping, share inner broadcast etc) but I still dont understand why I am seeing non-coherency

I have attached image of my vivado block design as well 

I have hit a dead end trying to solve this issue so will appreciate help

[kawazome]

I have some questions.

• What is the output value of the AxCACHE port?
• What is the output value of the AxPROT port?

Please refer to the following URL.

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842098/Zynq+UltraScale+MPSoC+Cache+Coherency 

> 9 Prototyping / Testing Coherency In The PL

[fahadak69]

I have used the same cache coherency wiki link you copied. Infact I have mentioned the same link in my post

Proc and cache values are as per description in the wiki. Axcache is 11 decimal and axprot is 2. I have confirmed from xilinx TRM that these values are correct

[kawazome · Accepted solution]

I presume that this issue is not a cache coherency issue between AXI-CDMA and PS (S_AXI_HPC0_FPD), but a cache coherency issue between BLK_MEM and PS (M_AXI_HPM0_FPD).

The cache is enabled when the program opens /dev/mem without O_SYNC and mmap (). So when PS executes an instruction to store data in BLK_MEM, the data is still in the cache and has not yet been written to BLK_MEM.
When he performs DMA in that state, the value of BLK_MEM before PS writes the data is transferred.

[fahadak69]

Yup that was the gap in my understanding. I failed to notice that direct readback from BRAM is via HPM and not HPC

Thanks alot!