UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Visitor hobbss1066
Visitor
4,567 Views
Registered: ‎01-10-2017

LTSSM Simulation stalling in Cfg Lnum Accept

Jump to solution

Running a simulation targeting a: Virtex 7 690T

 

IP built in Vivado 2015.4

 

In DUT: using AXI Bridge for PCI Express Gen3 Subsystem (2.0)

Basic mode, end point device.  x8, Gen2, 125MHz refclock, AXI  clock: 250MHz

 

In test bench: same IP (AXI Bridge for PCI Express Gen3 Subsystem (2.0)

Basic mode, Root Port of PCI Express Root Complex, x8, Gen 2, 125MHz refclock, AXI clock: 250MHz

 

I have the two PCIE devices (one in testbench, one in DUT -- FPGA device being tested) connected directly together.

 

At the beginning of the simulation, I wait until the axi_aresetn signal (sourced from the PCIE root port ip) goes high.

 

I then make the following writes to the Root Complex core (on the CTL axi port):

Address                  Value

0x00000018       0x00010100           --Set up sub bus, second bus, primary bus

0x00000068       0x00001020           --PCIE device control word

0x00000010       0x40000000           --Root EP BAR0 base address

0x00000150       0x20008000           --Root Port MSI Base Reg LSBits

0x00000004       0x00000007           --Command word - enable mem, io, and bus mastering from root

0x0000013C       0x00020000           --Enable MSI Interrupt received to cause an external interrupt

 

I then wait for the Root Port IP to enter state L0 -- which never happens (at least through 700 us).

 

Instead, the LTSSM state (as reported by reading 0x00000144) of the root port device, stalls at 0x10, or Cfg Lnum Acpt.

 

Looking in my wave window, I can see that the cfg_ltssm signal (of the root port) has gone through the following states:

00 - Det Quiet

01 - Det Quiet Gen2

02 - Det Active

04 - Pol Active

05 - Pol Config

06 - Pol Comp Pre Send Eios

08 - Pol Comp Send Pattern

07 - Pol Comp Pre Timeout

09 - Pol Comp Post Send Eios

0A - Pol Comp Post Timeout

10 - Cfg Lnum Acpt

0B - Cfg Lwidth St0

0D - Cfg Lwidth Ac0

0C - Cfg Lwidth St1 (long gap, link temporarily goes down)

0B - Cfg Lwdith St0    --> this is where my axi writes on the ctl port of the root complex start (see above)

0D - Cfg Lwdith Ac0

0E - Cfg Lwdth Ac1

10 - Cfg Lnum Acpt --> stays here indefinitely

 

The odd thing is that it looks like all link negotiation has been successfully completed (speed, number of lanes, etc.).  If I kick my simulation out of its wait state, and start using the PCIE link, it operates as expected (successfully read/write over the bus).

 

Additional fact: this FPGA design is working successfully in hardware.

 

Am I doing something wrong either in my simulation or in the configuration of my root port?

 

 

 

 

 

 

 

0 Kudos
1 Solution

Accepted Solutions
Highlighted
Xilinx Employee
Xilinx Employee
8,402 Views
Registered: ‎11-25-2015

Re: LTSSM Simulation stalling in Cfg Lnum Accept

Jump to solution

Hi @hobbss1066,

 

Yes please go ahead and mimic the routine i mentioned 

 

All your questions are related to PG054 which is applicable only for Gen2 Core

 

PG156( for Gen3 Base core) and PG194( for AXI PCIe Gen3 Core)

 

Moreover Virtex-7 and Ultrascale devices uses Gen3 Core and 7 series devices(except Virtex-7 ) uses Gen2 Core

 

Thanks,

Sethu

-----------------------------------------------------------------------------------------------

Please mark the post as "Accept as solution" if the information provided answers your query/resolves your issue.

Give Kudos to a post which you think is helpful.

 

 

 

0 Kudos
4 Replies
Xilinx Employee
Xilinx Employee
4,540 Views
Registered: ‎11-25-2015

Re: LTSSM Simulation stalling in Cfg Lnum Accept

Jump to solution

Hi @hobbss1066,

 

Can you cross-check the configuration sequence...I only see few lines from it in the post

 

Configuration of the AXI PCIe3 Bridge as in corresponding example design:  (Gen2 is almost similar too)

 

The configuration sequence appears both in m_axi_ctl_model.v and m_axi_model.v.

 

m_axi model – those are the writes to the various addresses for the tests.  These are going into the slave bridge of the RP part of the core, and are the data written to the Endpoint.

 

_ctl is the enumeration, _axi is the actual data interchange.

 

Most of these can be determined by mapping to the Memory Mapped section of the AXI PCIe document.  (Noting that these are root port)

 

These are ECAM accesses – so

[27:20] – Bus ---- 0x00 for Root port, 0x01 for Endpoint

[19:15] – Device Number – 0x0 for both

[14:12] – Function number – 0x0 for both (Exdes only does function 0)

[11:8] – Extended Register space – Access to extended configuration space

[7:2] – Register # in configuration space

[1:0] – Byte Address (Ignored for this set)

 

These are all configuration writes (no reads – it should be noted that this means we don’t actually check the size of the BAR requested… it’s a “feature” – holdover from the Gen2 core which was limited to a 32-bit BAR size). 

 

The following are both to the EP Type 0 configuration space. The first set is the address, the second is the data.  So for example:

set_add[2]  <= #TCQ 28'h0100010;

set_data[2]  <= #TCQ 32'hFFFFFFFF;

 

then

set_add[6]  <= #TCQ 28'h0100010;

set_data[6]  <= #TCQ 32'h80000000;

 

Write the mask, then set the BAR0 address 

 

The following is the most confusing one – but it is the dynamic setting of the AXIBAR2PCIEBAR_0L translation address for the Root port (without this, the example design won’t hit the endpoint BAR, particularly if the you changed it in the GUI).

 

Extended / Register – 0x20C – “AXI Base Address Translation Configuration Registers – offset 0x208 – 0x234) – which allows for the slave bridge of the RP model to send information

set_add[13] <= #TCQ 28'h000120C;

set_data[13] <= #TCQ 32'h80000000;

 

If you enumerated the BAR0 of the Endpoint to something other than 0x8000_0000, then you would also need to set the AXIBAR2PCIEBAR_0L of the RP to the same thing, or requests from the RP won’t hit the PCIe BAR on the EP.  Since the ExDes only does requests to BAR0, regardless of your selections, the other translations don’t have to get set.    If you are expanding this, then you would need to set the translation addresses appropriately.

 

You do have to use the combo of PG194 and PG156

 

So, in order:

      set_add[0]  <= #TCQ 28'h0000018;        RP – Set Secondary Lat Timer, Sub Bus Number, Secondary Bus Number, Primary Bus Number

      set_add[1]  <= #TCQ 28'h00000D4;       RP – Slot Cap

      set_add[2]  <= #TCQ 28'h0100010;       EP – BAR 0

      set_add[3]  <= #TCQ 28'h0100014;       EP – BAR 1

      set_add[4]  <= #TCQ 28'h0100018;       EP – BAR 2

      set_add[5]  <= #TCQ 28'h0100030;       EP – Expansion ROM Base

     set_add[6]  <= #TCQ 28'h0100010;        EP – BAR 0

      set_add[7]  <= #TCQ 28'h0100014;       EP – BAR1

      set_add[8]  <= #TCQ 28'h0100018;       EP – BAR 2

      set_add[9]  <= #TCQ 28'h0100030;       EP – Expansion ROM Base

      set_add[10] <= #TCQ 28'h0100070;      EP – (Not sure, should be reserved on the Gen3 cores)

      set_add[11] <= #TCQ 28'h0100004;      EP – Command Register

      set_add[12] <= #TCQ 28'h0001148;      RP – Memory Mapped RP Status/Control Register

      set_add[13] <= #TCQ 28'h000120C;      RP – AXI Base Address Translation Register

 

Not that this does not include:

1)      Setting PMCSR

2)      Bus Master Enable for EP if customer is going to use the slave bridge

3)      Setting up MSI

These are all expansions that you would have to do. 

 

Note:

 

0x01000070 – This is actually a carry-over from Gen2 – it should be 0x010000D0 for Ultrascale and Virtex-7 Gen3. This is the Link Control & Status register, obviously not absolutely required step.

 

On the RP “Function 1” – our root port only supports 1 function, so the function number in the ECAM – for the RP can be default set to 0. 

 

 

Thanks,

Sethu

-----------------------------------------------------------------------------------------------

Please mark the post as "Accept as solution" if the information provided answers your query/resolves your issue.

Give Kudos to a post which you think is helpful.

0 Kudos
Visitor hobbss1066
Visitor
4,522 Views
Registered: ‎01-10-2017

Re: LTSSM Simulation stalling in Cfg Lnum Accept

Jump to solution

Sethu,

 

Thank you for your response.  I will modify my setup routine to exactly mimic the one you pointed out from the example design.  However, I have a few questions:

 

1. 0x00000D4 --> you identify this as a RP - Slot Cap write.  However, looking at Table 2-21 in PG054 (September 30, 2015 -- page 42), Addresses 0xA8 - 0xFF are reserved.  The Slot Cap register is actually at 0x74.  Am I missing something?  In any case, is this register completely necessary?

 

2. For the last two writes (set_add[12] and set_add[13]), it looks like it is setting the Function to 001b --> i.e., the address is 0x0001148 instead of 0x0000148 and 0x000120C instead of 0x000020C.  Why?  Table 2-8 in PG194 (November 18, 2015 -- page 28), specifies that this is "Hard-wired to 0."

 

3. For the 10th write (i.e., set_add[10] <= 29'h01000070), you say that this should be reserved on the Gen3 core.  However, per Table 2-21 in PG054 (September 30, 2015 -- p. 41), address 0x070, part of the PCI Configuration Space Header, is Link Status and Link Control -- not "reserved".  The same table notes that 0xD0 IS reserved (part of the block from A8 to FF).  Again -- am I misinterpreting this?

 

After writing all this, I just compared PG054 (7 Series FPGAs Integrated Block for PCI Express v3.2) to PG156 (UltraScale Architecture Gen3 Integrated Block for PCI Express v4.0).  They have very different PCI Configuration Header Space memory maps.  I am targeting a Virtex 7 device with the AXI Bridge for PCI Express Gen3 Subsystem v2.0 (PG194).  Both your example (and my own auto-generated m_axi_ctl_model.v and m_axi_model.v files) seem to be using the address map in PG156, NOT the one in PG054 for the PCI Configuration space.  Why?  Is it because my core (PG194) is Gen3?  From the intro to PG156, it looks like that is only for Ultrascale devices (not 7 series).

 

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
8,403 Views
Registered: ‎11-25-2015

Re: LTSSM Simulation stalling in Cfg Lnum Accept

Jump to solution

Hi @hobbss1066,

 

Yes please go ahead and mimic the routine i mentioned 

 

All your questions are related to PG054 which is applicable only for Gen2 Core

 

PG156( for Gen3 Base core) and PG194( for AXI PCIe Gen3 Core)

 

Moreover Virtex-7 and Ultrascale devices uses Gen3 Core and 7 series devices(except Virtex-7 ) uses Gen2 Core

 

Thanks,

Sethu

-----------------------------------------------------------------------------------------------

Please mark the post as "Accept as solution" if the information provided answers your query/resolves your issue.

Give Kudos to a post which you think is helpful.

 

 

 

0 Kudos
Visitor hobbss1066
Visitor
4,501 Views
Registered: ‎01-10-2017

Re: LTSSM Simulation stalling in Cfg Lnum Accept

Jump to solution

So... it turns out I don't have a problem at all.

 

Looking at PG023 (rather than PG054), apparently the hex values of the various LTSSM states have changed values.

 

0x10 (the value where my simulation was "stalling") is actually L0 in the gen3 core.

 

This would explain why the core seemed to operate normally after I forced my sim to "ignore" that the IP wasn't in L0 (it actually was!).

 

Thanks for your help.

0 Kudos