we are having the following trouble with our Xilinx PCIe Endpoint design (Gen1, 1 Lane, XC7A35T): when the EP is put into power state D3 (by writing PMCSR to 3), the device is apparently going correctly into the D3 state (LinkState="L1 Idle", LTSSM=0x1A).
However on the PC side a 'PCIe correctable error' is detected by the upstream device (PCIe Device Status bit0 asserted). This results in a bluescreen with the current software setup (Win10). When the 'report correctable errors' is disabled (by clearing PCIe Device Control bit0), we get rid of the bluescreen.
The upstream device is an Intel C620 Root Port device (VenID=0x8086, DevID=0xA190) on a Supermicro X11SPL-F motherboard.
We only observe the issue on this particular motherboard and on that particular PCIe slot controlled by the C620.
In other slots no errors are detected by the upstream device when our EP goes into D3.
As a test I have put some totally different boards (Without Xilinx EP) into the problematic slot and put them manually into the D3 state, and here the root port is not detecting any errors. This somewhat suggests that the Xilinx EP might be exhibiting some compatibility issues when going into D3.
The problem remains the same when I use the example Xilinx EP PIO design (pci_7x_0_ex).
Before digging deeper into the investigation, I thought it would be useful to ask here first...
Investigation reveiled the following:
After the Xilinx EP has sent the last EIOS message (Electrical Idle Ordered Set = COM,IDL,IDL,IDL), it generates 7 more zero bits before going to electrical idle (both TX lines at the common mode voltage). [at least that is what I see in the simulation]
These 7 zerobits are probably reported by the upstream device as an invalid 8b10b symbol and causing the bluescreen.
The PCIe spec is not quite clear what is allowed to generate on the TX after sending the EIOS. The only requirement seems to be that the TX goes into electrical idle within 20 UI. Generating zero bits seems not forbidden, but of course generating a valid 8b10b symbol would be strategically better in order to avoid the reported issue.
For Gen1 (2.5Gbit/s) only 1 EIOS message needs to be sent.
For Gen2 (5Gbit/s) 2 EIOS messages are sent. I noticed the EP in Gen1 mode is also generating the 2nd EIOS message on the GTP's TXDATA bus, but at the same time it is suppressed by the asserted TXELECIDLE. As a test I patched the EP in such a way that the assertion of TXELECIDLE is delayed by 1 clk. In this way a valid COM-IDL sequence is generated on the TX after the first EIOS.
Now the Intel upstream device does not report any errors anymore when the device goes into D3 (and the Link into L1).
However this is not a good solution as it violates the 20 UI specification.
The ideal solution would be to generate 1 more valid 8b10b symbol and then going to electrical idle.
The GTP has a configuration parameter TX_EIDLE_ASSERT_DELAY. Perhaps this could be used?
On the other hand the Intel chip seems a bit too strict in reporting the issue. Strictly speaking the Intel port is still in the L0 state and not in the L1 state yet (this happens only when it has received the EIOS *and* it has self transmitted the EIOS which happens somewhat later) and in L0 is it allowed to report 8b10b errors. The question is if it makes sense to report such errors if you know that the transmitter is going into electrical idle within the first 20 UI.
In my opinion it would be an advantage for the Xilinx PCIe core if it could avoid the issue (and increase interoperability) by going into electrical Idle in a more graceful way.
Any comments appreciated from the Xilinx PCIe specialists !
it turns out that the GTP parameter TX_EIDLE_ASSERT_DELAY (undocumented in ug482) is set to 4 for simulation and to 2 for implementation. (see EI_DELAY in the core file pcie_7x_0_gt_top.vhd)
This parameter seems to influence when the TX line goes into electrical idle. (1 unit is one 8b10b symbol).
With the value set to 4, the EIOS is generated, immediately followed by 7 zerobits and next electrical idle. This is probably fine.
With the value set to 2, the line goes into electrical idle 20 UI earlier, hereby truncating the 2nd IDL symbol of the EIOS.
Could this be the root cause of the problem I see?
When I compile the FPGA with that parameter forced to 4, the issue of going into D3 seems to be solved.
Why is Xilinx using different settings for this parameter for simulation and implementation, and why is it set to 2? Is this a bug in the PCIe EP core? (Tested with Vivado 2018.2)
We have seen a handful of link partners that need the adjusted value of TX_EIDLE_ASSERT_DELAY due to an interop issue (generally resulting in a link downtrain). In this case the reporting of a correctable error in a state that certain behaviors should be expected, seems like less severe of a case.
That being said, a BSOD seems a bit of an extreme reaction at the OS to a Correctable Error (in this case, an 8b/10b decode error). Typically, in system design, we would expect BSOD/Kernel panics only on Uncorrectable or Fatal class errors, as a Correctable error is just that - corrected by the protocol. A receiver error, for example, could cause an 8b/10b decode or disparity error - which are very much allowable by the spec, within the BER.
do I understand correctly that what you are actually saying is:
- the correct value should be 4 in all cases (both for simulation and for implementation)
- because Xilinx has noticed some interop issues with a setting of 4, they deliberately configure it nowadays to 2 for implementation (resulting in transmission of a corrupted EIOS) for all designs because that overcomes in general most problems where links are not correctly trained to max performance (with the only drawback that this can cause some other, less critical issues, like in my case)
Since my EP only support 1 lane with 1 speed (2.5Gb/s), the design is probably less prone to false link training. Therefore using the correct setting of 4 would probably be the best choice for my design, or do you have more information that the setting is also causing link training issues for 1x Gen1 on some platforms?
It would be greatly appreciated if Xilinx could be a bit more open (e.g. Answer Record) on the interoperability issues they experience with the current 7-series PCIe EP IP core.
This was discussed internally. As Beth mentioned we had a handful of link partners which needed adjustment for TX_EIDLE_ASSERT_DELAY. With one particular vendor, we had observed that EIOS sequence was truncated before entering Electrical Idle. The full EIOS sequence consists of (BC-7C-7C-7C); the core was sending only (BC-7C-7C) and then enter Electrical Idle. PCI Express specification doesn't specify how many exact IDLs are required for EIOS definition; three or two IDLs (7C) can be considered a valid EIOS. However, that system was expecting three 7Cs. It sounds like this is what is happening in your case too. The TX_EIDLE_ASSERT_DELAY value of 3 resolved the issue in that system.
you are explaining exactly why the Xilinx EP is not working on all systems. But why is the Xilinx EP not corrected for this??
The PCIexpress specification (rev3) is 100% clear on this:
220.127.116.11. Electrical Idle Sequences
When using 8b/10b encoding, an EIOS is a K28.5 (COM) followed by three K28.3 (IDL) Symbols.
Transmitters must transmit all Symbols of an EIOS. An EIOS is received when the COM and two of the three IDL Symbols are received.
So as a transmitter (The Xilinx EP) shall always sent the complete sequence COM-IDL-IDL-IDL before going into Electrical Idle.
The receiving device could assume the EIOS received when it receives at least COM-IDL-IDL (but this is NOT allowing the transmitter to only send COM-IDL-IDL)
When I simulate the Xilinx EP with the implementation setting of TX_EIDLE_ASSERT_DELAY = 2, then only COM-IDL is sent which is clearly wrong behaviour (assuming that this simulation matches what happens in reality).
Setting it to 3, would generate COM-IDL-IDL which likely would work in most cases. But setting it to 4 seems the only correct setting. It is still not clear to me why Xilinx would deviate from this setting for implementation (and not for simulation)?
Perhaps this came into play when the datapath had another number of pipeline stages and Xilinx forgot to adapt the TX_EIDLE_ASSERT_DELAY when the number of pipeline stages changed??
Thanks for your explanation. This IP has been around for a while. Let me check on why different values for TX_EIDLE_ASSERT_DELAY was set for implementation and simulation. As far as I can tell, TX_EIDLE_ASSERT_DELAY set to '4' for hardware, based on the past experience, should be ok. However, let me check the details before confirming.