UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Visitor detho23
Visitor
890 Views
Registered: ‎08-20-2018

PCI extended configuration space not matching programming

Jump to solution

I'm working with the DMA/Bridge Subsystem for PCIe IP in AXI Bridge mode on a kcu116 board. Programming the board is fine, and when reading the PCIe config registers over JTAG using the JTAG to AXI IP, they are set correctly. However, when trying to read the PCIe config registers either through 'lspci' in Linux, or using 'pci_read_config_dword()' function in driver, the values of the registers are totally wrong.

 

The board is configured as an endpoint device and this is correctly reflected in the Header Type register (offset 0x0E) as the value is 0. The Vendor ID and Device ID (offsets 0x00 and 0x02 respectively) are also correct. It seems to be an issue with the extended configuration space registers. There are some values in the registers, but they are not correct (for example, the Bus Location register (offset 0x140) is filled in but does not match the port:bus:device.function numbers in 'lspci').

 

I'm referencing "pg213-pcie4-ultrascale-plus" and "pg194-axi-bridge-pcie-gen3". One other note is that while the device is configured as endpoint, I am able to successfully read/write to registers that I should not be able to, according to pg194 (for example, I can write a '1' bit to Root Port control register (offset 0x148)).

 

This seems like an AXI to PCIe translation issue since I can read over JTAG, but I don't understand why the low-address PCI registers would be presenting themselves on the bus fine when the extended space is getting messed up. Any help at all would be greatly appreciated.

0 Kudos
1 Solution

Accepted Solutions
Xilinx Employee
Xilinx Employee
892 Views
Registered: ‎12-10-2013

Re: PCI extended configuration space not matching programming

Jump to solution

Hi @detho23,

 

I think the confusion is coming from what is PCIe Configuration Space header information (available in LSPCI and via a direct configuration read), and what is only available from the s_axi_ctl BAR on the interface.

 

From the Table 2-4 in PG194 - only the registers from 0x000-0x12F will be available via Configuration Space.   This Configuration Space will be determined by the Next Cap pointers as laid out in PG213.

 

However, the registers starting at 0x130 and up are Bridge specific registers, and are only available via the s_axi_ctl BAR (S_AXIL) interface.

 

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
4 Replies
Xilinx Employee
Xilinx Employee
873 Views
Registered: ‎12-10-2013

Re: PCI extended configuration space not matching programming

Jump to solution

Hi @detho23,

 

Can you please provide the lspci -xxxvvv, your DMA/Bridge XCI, and confirm that you s_axi_ctl BAR is set to a 512MB aligned address in the AXI memory space?

 

 

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Visitor detho23
Visitor
867 Views
Registered: ‎08-20-2018

Re: PCI extended configuration space not matching programming

Jump to solution

Thanks for the response @bethe.

  • #lspci -xxxvvv output:

03:00.0 Memory controller: Xilinx Corporation Device 9038
Subsystem: Xilinx Corporation Device 0007
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 25
Region 0: Memory at e4000000 (32-bit, non-prefetchable) [size=32M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [60] MSI-X: Enable- Count=33 Masked-
Vector table: BAR=0 offset=00008000
PBA: BAR=0 offset=00008fe0
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.244W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [1c0 v1] #19
Kernel driver in use: PCIe-AXI
00: ee 10 38 90 06 00 10 00 00 00 80 05 10 00 00 00
10: 00 00 00 e4 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 ee 10 07 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 0b 01 00 00
40: 01 60 03 00 08 00 00 00 05 60 80 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 11 70 20 00 00 80 00 00 e0 8f 00 00 00 00 00 00
70: 10 00 02 00 23 80 d0 0f 20 58 09 00 83 f0 43 00
80: 40 00 83 10 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 16 00 00 00 00 00 00 00 0e 00 00 00
a0: 03 00 1e 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

 

  • I'm not entirely sure which .xci file to include so I've attached all 5 that were generated.
  • As far as s_axi_ctl BAR, I have the following set in my Address Table:
  • CellSlave InterfaceBase NameOffset AddressRangeHigh Address
    axi_bram_ctrl_0S_AXIMem00xC000_00008K0xC000_1FFF
    axi_bram_ctrl_1S_AXIMem00xC200_00008K0xC200_1FFF
    xdma_0S_AXI_BBAR00x7600_00001M0x760F_FFFF
    xdma_0S_AXI_LITECTL00x0000_00004K0x0000_0FFF

If you are referring to the CTL0 memory space, I will try setting it to 512M aligned and see if that is the issue.

 

I should also mention that this AXI Bridge Subsystem for PCI Express core is part of a block design with other cores attached, not sure if that is helpful.

0 Kudos
Xilinx Employee
Xilinx Employee
893 Views
Registered: ‎12-10-2013

Re: PCI extended configuration space not matching programming

Jump to solution

Hi @detho23,

 

I think the confusion is coming from what is PCIe Configuration Space header information (available in LSPCI and via a direct configuration read), and what is only available from the s_axi_ctl BAR on the interface.

 

From the Table 2-4 in PG194 - only the registers from 0x000-0x12F will be available via Configuration Space.   This Configuration Space will be determined by the Next Cap pointers as laid out in PG213.

 

However, the registers starting at 0x130 and up are Bridge specific registers, and are only available via the s_axi_ctl BAR (S_AXIL) interface.

 

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
Visitor detho23
Visitor
827 Views
Registered: ‎08-20-2018

Re: PCI extended configuration space not matching programming

Jump to solution

Yes that clears up my confusion, thank you for your help.

0 Kudos