UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Observer fransschreuder
Observer
2,969 Views
Registered: ‎08-12-2010

PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

Hello,

Until recently, all the datasheets and product tables showed support for PCIe Gen3x16 and Gen4x8. A recent update shows that Gen4x8 is now only supported for the Ultrascale+ devices with HBM2 technology, for the other devices gen4 has magically disappeared.

In vivado the ip core has a hidden option to enable gen4 support which I am not allowed tho show here, but you can pm me to get the tcl line.

What is the reason for silently removing gen4 support? Was there a problem with the gen4 compliance? Did it not work in some cases?

I would like to know whether we can use gen4 or that this is a road with a dead end, we have already designed our hardware taking gen4 into account and we don't want to throw it all in the bin.

1 Solution

Accepted Solutions
Xilinx Employee
Xilinx Employee
2,753 Views
Registered: ‎09-02-2009

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

Hi Frans - sorry for the confusion.  I'm the marketing manager for PCIe.  Let me try to clarify.

When our UltraScale+ device first taped out, the PCISIG 4.0 spec was only at v0.5.  That was the spec that our first PCIe blocks used.  There were two main changes going from v0.5 to v0.7 (and subsequent versions all the way to 1.0).  First, there was an updated Electrical Idle Ordered Set and second there was a requirement for a new CTRL SKP Ordered Set.  Because of Xilinx's deep involvement in PCISIG, we knew that the Electrical Idle OS was coming and added this as option in silicon.  However, the CTRL SKP Ordered Set spec work did not even start until after we taped out.

At the time our devices started sampling, there were some devices (including Power 9 from IBM) that had an option for v0.5 (with the updated EIEOS).  Because of the 'stamp and repeat' nature of the way we build our devices, all UltraScale+ devices were built with this original PCIe block (called PCIE4 in hardware).

As time moved on, all devices that we know of moved away from having a v0.5 option.  Also, our testing had shown that the new CTRL SKP caused our UltraScale+ devices not to link train.  When we introduced the new UltraScale+ HBM devices, that included a spin to the PCIe block to add CCIX.  When we added CCIX, we took the opportunity to update the PCIe block to have support for the new CTRL SKP OS as well.  We call this block PCIE4C to differentiate it from the original PCIE4 blocks in UltraScale.  We have found that this new PCIE4C block, while not fully compliant, works at Gen4 with every Gen4 devices we have tested with so far (which is quite a few devices from different vendors).

What we have found though is that while our PCIE4 devices only work with other v0.5 devices, many people miss that note.  And because we don't see any v0.5 devices in the market anymore, we decided that it was causing more confusion and thus decided to remove the Gen4 designation from the datasheet.  It's also worth noting that the option you put above will be removed from Vivado and no longer work.

Just to recap:

1. Xilinx UltraScale+ devices PCIe block supported 4.0 v0.5.  There are no other v0.5 devices anymore.  Causes confusion and being removed.

2. Xilinx UltraScale+ HBM devices have a new PCIE4C block that is compatible to Gen4 4.0 and interoperates with other 4.0 devices even though not fully compliant.  A list of limitations can be found in PG213.  (Also note that some HBM devices have both PCIE4 and PCIE4C blocks).

3. Xilinx Versal devices (7nm) will be fully compliant to the 4.0 PCIe spec.

Sorry for the long winded explanation but hopefully this provides you some context and helps you choose the right device for your application.

Jason  

 

 

18 Replies
Observer fransschreuder
Observer
2,943 Views
Registered: ‎08-12-2010

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

I do realize that the Ultrascale+ devices were released before the PCIe Gen4 standard became official. I can also understand that because of this the PCIe hard block has limitations regardsing PCIe Gen4.

However I don't think it is fair to just remove all the gen4 from the datasheets as if it had never been there and just be silent about it. Please make a document with the limitations and conditions under which the Gen4 hard block in the lower range Virtex Ultrascale+ devices can be used if any.

0 Kudos
Moderator
Moderator
2,915 Views
Registered: ‎02-11-2014

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

Hello @fransschreuder,

The PCIE4 hardblocks were designed for PCIe gen4 spec 0.5+ (which as the time was the currently used market standard.) The current market now is gen4 spec 0.7 - 1.0. Quite a bit changed between the 0.5 spec and the 1.0 spec. Some features that were originally optional in the 0.5 spec, became required in the 1.0 spec. The hidden gen4 switch was never publicly documented for the PCIE4 hard blocks for the above reasoning.

As you have stated the PCIE4C hardblocks (that you can find in HBM devices) DO support some gen4 spec 1.0 features (that the PCIE4 hard blocks do not support) which can be seen in the following Table from PG213. Take special consideration of the IMPORTANT note at the bottom. These features allow our PCIE4C hard blocks to interoperate with other gen4 devices, but we are not gen4 compliant.

image.png

We have also updated DS890 to reflect gen4 support which can be seen here:

image.png

Thanks,
Cory

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
Observer fransschreuder
Observer
2,901 Views
Registered: ‎08-12-2010

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

Dear @coryb,

Thank you for the answer. I would like some more clarification though:

You state that the PCIE4 blocks were Gen4 0.5+ compliant, but PG213 states that the PCIE4C block is compliant to Gen4 0.5.

I would like to see a list of limitations regarding the PCIE4 blocks with respect to PCIe Gen4 1.0, rather than just removing every reference from the datasheet.

Additionally we will test our Gen4 design with the VU9P FPGA in an IBM power9 server to actually experiment with Gen4. If we don't get information from Xilinx we have to find it out for ourselves.

Regards,

Frans

0 Kudos
Observer fransschreuder
Observer
2,789 Views
Registered: ‎08-12-2010

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

@coryba comment would be highly appreciated.

0 Kudos
Xilinx Employee
Xilinx Employee
2,754 Views
Registered: ‎09-02-2009

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

Hi Frans - sorry for the confusion.  I'm the marketing manager for PCIe.  Let me try to clarify.

When our UltraScale+ device first taped out, the PCISIG 4.0 spec was only at v0.5.  That was the spec that our first PCIe blocks used.  There were two main changes going from v0.5 to v0.7 (and subsequent versions all the way to 1.0).  First, there was an updated Electrical Idle Ordered Set and second there was a requirement for a new CTRL SKP Ordered Set.  Because of Xilinx's deep involvement in PCISIG, we knew that the Electrical Idle OS was coming and added this as option in silicon.  However, the CTRL SKP Ordered Set spec work did not even start until after we taped out.

At the time our devices started sampling, there were some devices (including Power 9 from IBM) that had an option for v0.5 (with the updated EIEOS).  Because of the 'stamp and repeat' nature of the way we build our devices, all UltraScale+ devices were built with this original PCIe block (called PCIE4 in hardware).

As time moved on, all devices that we know of moved away from having a v0.5 option.  Also, our testing had shown that the new CTRL SKP caused our UltraScale+ devices not to link train.  When we introduced the new UltraScale+ HBM devices, that included a spin to the PCIe block to add CCIX.  When we added CCIX, we took the opportunity to update the PCIe block to have support for the new CTRL SKP OS as well.  We call this block PCIE4C to differentiate it from the original PCIE4 blocks in UltraScale.  We have found that this new PCIE4C block, while not fully compliant, works at Gen4 with every Gen4 devices we have tested with so far (which is quite a few devices from different vendors).

What we have found though is that while our PCIE4 devices only work with other v0.5 devices, many people miss that note.  And because we don't see any v0.5 devices in the market anymore, we decided that it was causing more confusion and thus decided to remove the Gen4 designation from the datasheet.  It's also worth noting that the option you put above will be removed from Vivado and no longer work.

Just to recap:

1. Xilinx UltraScale+ devices PCIe block supported 4.0 v0.5.  There are no other v0.5 devices anymore.  Causes confusion and being removed.

2. Xilinx UltraScale+ HBM devices have a new PCIE4C block that is compatible to Gen4 4.0 and interoperates with other 4.0 devices even though not fully compliant.  A list of limitations can be found in PG213.  (Also note that some HBM devices have both PCIE4 and PCIE4C blocks).

3. Xilinx Versal devices (7nm) will be fully compliant to the 4.0 PCIe spec.

Sorry for the long winded explanation but hopefully this provides you some context and helps you choose the right device for your application.

Jason  

 

 

Observer fransschreuder
Observer
2,726 Views
Registered: ‎08-12-2010

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

Dear @jlawley,

Thanks for the detailed explanation. This was exactly what I was looking for and now we have at least a proper foundation on which we can base our further design decisions. You have to understand that if you specify Gen4, people already start designing with it, then if it gets removed without notice or explanation, people get pissed off.

I think that a note similar to your answer belongs in the PG213 document, and it is a pity that you are going to remove the hidden enable option from Vivado. Luckily we can choose to keep using the current version of Vivado if we decide to go on with this development and the IBM power9 device or something similar with Gen 4.0 v0.5 support.

0 Kudos
Scholar samcossais
Scholar
2,325 Views
Registered: ‎12-07-2009

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

@jlawley Hi Jason, thank you for you explanation.

Gen4 support has been indeed a pretty confusing point. There is a lot of documentation and it was hard to not miss the mention you might have done about "old" Ultrascale devices compliant with a deprecated and quite unusable version of PCIe Gen4. I understand your points but I suggest that you improve communication, it was at least ambiguous for most of us to be honest.

Even now with the Ultrascale+ HBM devices, the PCIE4C block is supposed to have solved the problem but I still can't select the Gen4 speed. I am using the Ultrascale+ PCI Express 4c Integrated Block (1.0), Vivado 2018.3 and a Ultrascale+ HBM, namely the XCVU37P-FSVH2892-2E, as we plan to get a VCU128 board. For maximum link speed with x8 lane configured, only speeds up to Gen3 are selectable, no matter what block I choose. I checked PG213 page 282 and it's written "Note: A Gen4 speed option will be available in a future release.", which is a quite important point for a side note.

I know you usually don't like to give clear dates for future releases but things have been already confusing enough so it would be nice if you were able to tell us when this future release including Gen4 support is expected to occur ? At least a time frame would be nice.

 

Best regards,

Sam

Xilinx Employee
Xilinx Employee
2,283 Views
Registered: ‎09-02-2009

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

Hi Sam - We expect to make the the Gen4 option public with 2019.1.  We've been continuing to test our solution both with partners wo will also offer Gen4 PCIe solutions and by attending PCIe plugfests to ensure that our block will interoperate with other Gen4 solutions.  I don't have exact dates for 2019.1, but usually the .1 software releases from Xilinx happen end of Q1 or beginning of Q2.

Jason

Scholar samcossais
Scholar
2,268 Views
Registered: ‎12-07-2009

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

Thank you Jason for your clear reply. Much appreciated.

Sam

0 Kudos
Scholar samcossais
Scholar
1,958 Views
Registered: ‎12-07-2009

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

Hi,

sorry for digging this topic out again but I was wondering if all the Xilinx PCIe IP will be concerned by the Gen4 upgrade (see image) ?

xilinx_pcie_ip.PNG

I use the "DMA/Bridge Subsystem for PCIe" IP as a root port and in AXI bridge mode and I want to test it in Gen4 asap. If this option comes too late for this IP, I might consider making my own AXI bridge and use the "UltraScale+ PCIe 4c Integrated Block" IP, although this would require me more work.

0 Kudos
Scholar samcossais
Scholar
1,494 Views
Registered: ‎12-07-2009

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

Hi,

I have just installed Vivado 2019.1, upgraded all my IPs in my current project, but still no setting available for Gen4 in the "DMA/Bridge Subsystem for PCIe" IP. This is really disappointing to say the least as I have been waiting for it for a few months already.

When will the Gen4 version of this IP be available ?

0 Kudos
Scholar samcossais
Scholar
1,359 Views
Registered: ‎12-07-2009

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution
Still no information about the "DMA/Bridge Subsystem for PCIe" IP ? I am really disappointed to say the least...
0 Kudos
Xilinx Employee
Xilinx Employee
1,355 Views
Registered: ‎09-02-2009

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

Sorry for the delay in getting back to you.  Looks like it was an oversight on our part and the code didn't properly make the Gen4 option visible.  We're in the process of setting up a patch.  Please pm me your info and I'll make sure you get access to the patch ASAP.

0 Kudos
Scholar samcossais
Scholar
1,289 Views
Registered: ‎12-07-2009

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

Ok thank you. Any time frame for the patch ? (I guess before 2019.2 comes out ?)

0 Kudos
Observer fransschreuder
Observer
1,277 Views
Registered: ‎08-12-2010

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

I saw today that the PCIE4C devices show the Gen4 uption in Vivado 2019.1, but you have to choose the pcie4c ip core

0 Kudos
Scholar samcossais
Scholar
1,272 Views
Registered: ‎12-07-2009

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

Yes for the PCIe Integrated Block IP but not for the DMA/Bridge IP, which includes the PCIe Block IP but with other logic as well. And that is my point (reread my comments). I also knew there was a workaround to enable Gen4 for the block IP with 2018.3, my point was that it wasn't possible to do that for the DMA/Bridge IP, and it's still not with 2019.1

0 Kudos
Scholar samcossais
Scholar
923 Views
Registered: ‎12-07-2009

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

I have just tested Vivado 2019.1.1 and finally the PCIe DMA/Bridge IP has the Gen4 option available in an official version.

Which means finally all the Xilinx IPs using PCIe now offer Gen4 support when using Virtex Ultrascale+ HBM devices and PCIE4C blocks.

End of the story !

Newbie walshcat
Newbie
317 Views
Registered: ‎08-20-2015

Re: PCIe Gen4 x8 support for Virtex UltraScale+ devices

Jump to solution

Hello, I read through this forum thread and had a couple questions,  when using U280 in a system, and link does not go up for PCIE 4, is there a specific setting to set it for x8 so the link will go "up" or is this not possible?

0 Kudos