cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
fshenmi
Contributor
Contributor
751 Views
Registered: ‎03-11-2016

PSU-PCIe master AXI failed to access PS-DDR memory space

Jump to solution

Hi,

I'm porting a baremetal NVMe driver from the PL XDMA to the PSU PCIe. Both in Root Complex mode. The idea is pretty simple. Changing the base address of AXI slave port to the BAR offsets on the PSU PCIe. The driver only runs after successful end point enumeration of SSD controller (Xilinx PSU baremetal example).

A53 can access the SSD controller register freely (both read and write) through 0x600000000 16KiB BAR0.

However it appear the SSD controller cannot access the main DDR4 memory space through its hardened AXI3 Master port.

Once I issue the first Admin command the SSD controller will timeout with a "Received Master Abort" error in the ECAM device status register.

mrd -size d 0x8000100000 // ECAM Device Vendor ID/STS

0010000682011CC1

// right after the first admin command

2018000682011CC1

I've checked the I_ISUB_CONTROL bit 0 to be in enabled state thus any ingress traffic should pass through.

I've also tried enabling a single TRAN_INGRESS_CONTROL == 0x00130001 to allow ingress translation to access the lower 2GiB DDR memory space where my NVMe Admin/Completion queue resides.

This issue is similar to PL XDMA without PCIe:BAR PF_BAR) enabled when AXI-M access is blocked.

https://forums.xilinx.com/t5/PCIe-and-CPM/ZC706-NVMe-SSD-quot-Received-Master-Abort-quot-error/td-p/934030

Tags (1)
0 Kudos
1 Solution

Accepted Solutions
fshenmi
Contributor
Contributor
403 Views
Registered: ‎03-11-2016

Problem solved.

The default baremetal driver did not enable the memory space and bus mastering on the Root Complex. This is required for EP->RC ingress translations.

Xil_Out8(0x8000000004, 0x6); // ECAM rootport command register

Notice the driver correctly enable both for the Endpoint though.

 

Attaching my test result for XPG S11 pro SSD.

NVMe initialization successful. PCIe link is Gen2 x4.

5s delay for SSD...
Writing 16384 blocks of 262144 bytes...
Write Speed: 1528 MiB/s
10s delay for SSD...
Reading 16384 blocks of 262144 bytes...
Read Speed: 1696 MiB/s
Writing 32768 blocks of 131072 bytes...
Write Speed: 1490 MiB/s
10s delay for SSD...
Reading 32768 blocks of 131072 bytes...
Read Speed: 1696 MiB/s

View solution in original post

5 Replies
garethc
Moderator
Moderator
645 Views
Registered: ‎06-29-2011

Hi @fshenmi 

Have you considered the example drivers located in the Xilinx Github for PSU PCIE?

https://github.com/Xilinx/embeddedsw/tree/master/XilinxProcessorIPLib/drivers/pciepsu/examples

 

Thanks,

Gareth


------------------------------------------------------------------------------------------------

Don’t forget to reply, kudo, and accept as solution.

If starting with Versal take a look at our Versal Design Process Hub and our
Versal Blogs

------------------------------------------------------------------------------------------------
0 Kudos
fshenmi
Contributor
Contributor
624 Views
Registered: ‎03-11-2016

Hi Gareth,

Yes, I ran xpciepsu_rc_enumerate_example.c first. Only when that routine succeeds then I ran my NVMe driver.

The behavior is consistant with a failed PCIe master to AXI-M port access.

 

Thanks!

0 Kudos
fshenmi
Contributor
Contributor
542 Views
Registered: ‎03-11-2016
bump
0 Kudos
fshenmi
Contributor
Contributor
460 Views
Registered: ‎03-11-2016

I've tried adding AXI-BRAM and system ILA on the bus to monitor the transactions.

The first Admin queue was migrated to this AXI-BRAM.

Turns out CPU can access AXI-BRAM. But PSU PCIe cannot access it.

Is there anything else blocking this master port?

0 Kudos
fshenmi
Contributor
Contributor
404 Views
Registered: ‎03-11-2016

Problem solved.

The default baremetal driver did not enable the memory space and bus mastering on the Root Complex. This is required for EP->RC ingress translations.

Xil_Out8(0x8000000004, 0x6); // ECAM rootport command register

Notice the driver correctly enable both for the Endpoint though.

 

Attaching my test result for XPG S11 pro SSD.

NVMe initialization successful. PCIe link is Gen2 x4.

5s delay for SSD...
Writing 16384 blocks of 262144 bytes...
Write Speed: 1528 MiB/s
10s delay for SSD...
Reading 16384 blocks of 262144 bytes...
Read Speed: 1696 MiB/s
Writing 32768 blocks of 131072 bytes...
Write Speed: 1490 MiB/s
10s delay for SSD...
Reading 32768 blocks of 131072 bytes...
Read Speed: 1696 MiB/s

View solution in original post