cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Visitor
Visitor
1,028 Views
Registered: ‎11-21-2019

64-bit SDRAM access from PL on ZCU-102 board reads wrong value at every other address

Related original topic here https://forums.xilinx.com/t5/Xilinx-IP-Catalog/Block-memory-64bit-wide-skips-every-other-value/m-p/777290  in which the user was having the same problem reading from block memory.

The answers to that problem (never addressed or approved by a Xilinx employee) were:

"The AXI HPM0 FPD Data Width in the Zynq PS block must be set to 128 bit, even if you are only going to connect 64 bit wide memories to it."

and

"I solved this problem with a similar workaround setting the port width to 128 and adding a width converter."

 

In my case, I have bare VHDL code implementing an AXI master in the PL, which interfaces to the S_AXI_HP0_FDP slave port on the PS, to access 64-bit SDRAM on the ZCU102 board. There are no Xilinx AXI IP blocks (interfaces or width converters) in my PL.

Similar to the previous post, my setup correctly reads only the words in location for which address ends with 0.


For example, at addresses
0x40000000 -> reads ok
0x40000008 -> not ok (returns the data from 0x40000000)
0x40000010 -> reads ok
0x40000018 -> not ok (returns the data from 0x40000010)
...
If I try to read the content of a location ending with 8, I get the content of the previous location. (Addresses increment by 8 with 64 bit storage, and I've used a short C program in the PS to fill every 64-bit memory location with a value equal to its address, so I can see by the return value which address my logic in the PL actually read.)

Since I'm using bare VHDL code to drive the AXI ports, I was using ARSIZE[2:0] = "011" to access 64 bits at a time. With the S_AXI_HP0_FDP peripheral reconfigured for 128-bit data width, I rebuilt my PL first with ARSIZE[2:0] = "011" and then again with ARSIZE[2:0] = "100" to access 128 bits at a time. In both cases, I get the same result. All my reads are "bursts" of length = 1; i.e., a new, aligned address cycle precedes every read data cycle. Every other read is correct.

What's going on??? Even if the previous answers can be made to work, I'll need some explanation of "why" for my safety regulators. But really, those answers don't make any sense to me at the moment.

What could be causing reads from address xxxxxxx8 to return the data from xxxxxxx0?

---------

Related question: I'm doing all this debug with an ILA on the ZCU102 target. Is there a simulation model (BFM) for the S_AXI_HP0_FDP port that will provide realistic responses to the transactions I'm driving it with?

 

Thanks very much!!!

 

0 Kudos
Reply
9 Replies
Visitor
Visitor
953 Views
Registered: ‎11-21-2019

I found another possibly-related thread here:

https://forums.xilinx.com/t5/Memory-Interfaces-and-NoC/AXI-4-addressing-read-command/td-p/665882 

but it seems to only document an issue rather than resolve it:

That user appears to be using a MIG core in a 32-bit Zynq, whereas I am using the "hard" DDR controller on a 64-bit Ultrascale+. However, his observation that the bottom 2 read address bits are being masked is similar to my case of the bottom 4 address bits being masked.

The answer includes a table showing the relationship between PHY-to-MC clock ratio, UI Data Width, Memory Interface Data Width, and the resultant AXI Byte Address Masking. But I can't make the logical leap to how I change anything in my environment to actually read addresses that end with "8".

FWIW, my PS side is running OS Type "standalone" (v6.8) in 64-bit mode, and all the memory reads/writes from the PS side can be viewed as containing correct data in the SDK memory monitor window.

Help??? :)

 

0 Kudos
Reply
Adventurer
Adventurer
915 Views
Registered: ‎10-17-2016

Hi,

I was pointed here by @jmjohns8 who has found my question regarding a very similar problem.

In my case, the solution was to update the hardware description in the petalinux project. I had made changes to the bus width, but had forgotten to update the hardware description. This caused an incorrect bus width setting in the device tree.

Hope this helps,

Tobias

0 Kudos
Reply
Xilinx Employee
Xilinx Employee
866 Views
Registered: ‎10-04-2016

Hi @jmjohns8 ,

What is the AXI data width on your PL master IP?  What is the data width that you have set in the MPSoC configuration GUI for the S_AXI_HP0_FDP? It sounds like you have changed this a few times, but I'm not sure what combinations you have tried together. Please make sure that hte data width of your IP matches the data width of the S_AXI_HP0_FPD interface. The description of your problem sounds like there is a mismatch.

Do you have an ILA trace of the AXI command from your PL master going into the PS?

There is a simulation model for the AXI Interfaces in the Zynq PS. Please see DS941 for details about the API. If you open the base ZUS+ example design in Vivado, a basic test bench is available with that project.

https://www.xilinx.com/support/documentation/ip_documentation/zynq_ultra_ps_e_vip/v1_0/ds941-zynq-ultra-ps-e-vip.pdf

Regards,

Deanna

 

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Reply
Visitor
Visitor
837 Views
Registered: ‎11-21-2019

welo_zhaw, thank you for the information. I'm running the "standalone" OS instead of PetaLinux, so I'm not sure if there is an equivalent device tree or other place to set that configuration. Is there another way to control these settings? It seems like they should all be part of the Hardware Definition File when I generate output products.

Deanna, thanks also for your reply. I'm debugging some write issues at the moment, and will have to re-create my read-only build in order to snapshot the ILA traces for you. Coming soon...

In short, my PL TOP top-level is user-coded VHDL, with my own "custom, ultra-lightweight DMA IP" as a VHDL subblock below this top. My PS logic is all in a GUI block design. I "generate output products" for the GUI BD every time I change a parameter, and instantiate that in my top level. So my "lite DMA IP" and the BD are both components at the top level, connected to each other under my VHDL TOP. My ZCU102 board has a 1GB x 72 DDR4 DIMM (64-bit + ECC).

I've tried 3 cases (width = 32, 64, 128), and in each case, my VHDL IP and the S_AXI_HP0_FPD bus width are adjusted to match each other.

I'm pre-filling DDR4 with fixed patterns from software, and then reading that range of DDR4 from my VHDL PL, and saving each value read into a capture register whose width matches the bus width.

Each read operation is only one beat long (ARLEN = 0), but they are still supposed to be full AXI4 transactions. (Even though they look kind of like like AXI-Lite, the one-beat xfers are just placeholders until I enable longer bursts.)

In 128-bit mode (16 bytes), I capture all the read values I expect. The AXI address bus value is incremented by 16 to match the number of bytes read.

In 64-bit (8-byte) mode, AXI addresses increment by 8. I see consecutive reads being performed, and all appear to be valid AXI transactions, with ReadResponse = "00". When the address ends with hex 0 (e.g., 0x4020FFF0), the correct value is copied into my capture register. When my Read_FSM goes thru the same sequence of states for an address ending with hex 8 (e.g., 0x4020FFF8), the capture register does not change, and neither does the AXI read data bus. Despite attempting to read address 0x4020FFF8, it is returning the value stored at 0x4020FFF0.

When I run with my AXI "DMA" IP and port S_AXI_HP0_FPD configured for 32 bits (4 bytes), AXI addresses are incremented by 4. I get similar behavior to the 64-bit case, but now only every 4th read (from address ending with hex 0) is good, and reads from addresses ending in hex {4,8,C} still put the data from address ending in hex 0 into my capture register.

A colleague also suggested I check the register RDCTRL (AFIFM, in particular address 0xFD380000 to target S_AXI_HP0_FPD ). It was always set to its default value of 0x3B0, which is described as configuring the Read Channel Fabric Interface Width to 128 bits. It was running that way when my 128-bit build was working perfectly. I changed it to 0x3B1 (64 bits) while testing the 128-bit build, and I did not see any change in my waveforms. I'm a little confused about exactly what this register does, and why we would ever set this register from software; shouldn't that be part of the Hardware Definition File after I customize the PS/PL interface on my block design and generate output products?

Thanks again,

MarkJ (jmjohns8)

 

0 Kudos
Reply
Xilinx Employee
Xilinx Employee
827 Views
Registered: ‎10-04-2016

Hi @jmjohns8 ,

If the RDCTRL.FABRIC_WIDTH and WRCTRL.FABRIC_WIDTH fields don't match your targeted data width, you'll definitely see the behaviour you describe. So if your data width is 64-bits, these fields need to be 2'b01 to indicate 64-bits.

Which version of the tools are you running? In the 2016.x time frame, we had troubles with these settings:

https://www.xilinx.com/support/answers/66295.html

I just ran experiments last week in 2019.1 with the MPSoC PS in the IPI generated top level wrapper and did not see issues with the settings of these registers. As I changed the data width of the various slave interfaces in the MPSoC configuration GUI, the correct settings were reflected in psu_init.c.

Could you elaborate a bit more on the structure of your project? I want to better understand this comment:

In short, my PL TOP top-level is user-coded VHDL, with my own "custom, ultra-lightweight DMA IP" as a VHDL subblock below this top. My PS logic is all in a GUI block design. I "generate output products" for the GUI BD every time I change a parameter, and instantiate that in my top level. So my "lite DMA IP" and the BD are both components at the top level, connected to each other under my VHDL TOP. My ZCU102 board has a 1GB x 72 DDR4 DIMM (64-bit + ECC).

Where do you instantiate the PS in your design?

Regards,

Deanna

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
Visitor
Visitor
728 Views
Registered: ‎11-21-2019

Hi @demarco ,

I'm using Vivado and SDK v2018.3

I've attached a PDF file with a few slides that I hope will clarify the hierarchy.

I'm still a little confused about the RDCTRL.FABRIC_WIDTH and WRCTRL.FABRIC_WIDTH fields. I thought those would be hardcoded by the tool to values matching my "Customize Block Design" step, and exported when I "Generate Output Products" including the HDF file. Should the software boot-up phase execute those register settings automatically before my user C code starts to run? Can modifications I make to these register fields dynamically change the fabric width one the device is configured and running? It seems like those would be fixed at configuration time.

Thanks again,

jmjohns8

0 Kudos
Reply
Visitor
Visitor
765 Views
Registered: ‎11-21-2019

Hi @demarco ,

I am running v2018.3 of Vivado and SDK. Compiling and customizing top-level and block design on Linux, running SDK and HW Mgr on Windows 10.

I've attached a .PDF file that should help clarify the hierarchy.

I'm still a bit confused about the RDCTRL.FABRIC_WIDTH and WRCTRL.FABRIC_WIDTH fields. I thought that when I customized the PS instance with specific widths for the AXI Slave ports, and generated output products, that a Hardware Definition File or related configuration file would pass this information to the boot-up process.

Are these registers supposed to be written with non-default, user-specific values during boot-up? If so, where does that process get the width information from?

Is it valid for user application code to update these registers? It seems like they should be a static, hard configuration for all time after boot-up/configuration. In fact, it seems odd that they would even be modifiable by software, if they really specify a hardware-only configuration.

 

0 Kudos
Reply
Xilinx Employee
Xilinx Employee
752 Views
Registered: ‎10-04-2016

Hi @jmjohns8 ,

The RDCTRL.FABRIC_WIDTH and WRCTRL.FABRIC_WIDTH fields are set in psu_init.c or psu_init.tcl. If you unzip an HDF file, these files are inside of it. The settings for RDCTRL.FABRIC_WIDTH and WRCTRL.FABRIC_WIDTH should be set based on the configuration GUI for MPSoC PS.

Just as a sanity check, please search for the RDCTRL/WRCTRL registers on all enabled slave AXI interfaces in psu_init.c/psu_init.tcl and verify that there is a mismatch between what you are setting in the configuration GUI and what is in those files.

Here's the register definitions and addresses:

https://www.xilinx.com/html_docs/registers/ug1087/hh_goto.htm#afifm___rdctrl.html

https://www.xilinx.com/html_docs/registers/ug1087/hh_goto.htm#afifm___wrctrl.html

The psu_init.c file is incorporated into FSBL and run when the MPSoC device boots. If you are using a JTAG driven flow through SDK, psu_init.tcl performs similar functions to initialize the system.

It's not valid for a user application to update these registers. They should be static.

Thanks for providing your design hierarchy. I'm trying a few experiments based on that information and your Vivado version + OS.

Regards,

Deanna

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
568 Views
Registered: ‎02-07-2019

This saved me a bunch of debugging time, thank you! I just recently switched over to Vitis 2019.2 and am still getting used to the new workflow for updating the hardware specification. In my generated platform for a Zynq Ultrascale+, it seems like psu_init.c and psui_init.tcl have the correct RDCTRL.FABRIC_WIDTH in the [platform]/hw but not in [platform]/zynqmp_fsbl, which is the one being used in my project. Not sure if this is a bug or something I did incorrectly. But I'm glad to know what to look for now.

0 Kudos
Reply