Multi FPGA Virtex 6 communication using PCIe switch

venkat_vs2k2 · ‎11-19-2010

Hello all,

I am not sure if this is the right place to ask this question.

But I am trying to realize a peer-to-peer communication protocol between two V6 FPGAs using a PCIe Gen2 switch on board. I have a general idea on how PCIe host to FPGA communication is done. But i really dont find much about multi-fpga peer-to-peer communication using PCIe switch.

However many of my searches resulted in multi-host communication using PCIe swtiches(from different switch vendors) using a feature called Non-transparent mode. But not much information on multiple PCIe endpoint communication . Could someone please give some information on this ?

Thank You,

Venkat.

barco2 · ‎02-13-2009

Why would you want to do this? For a simple point-to-point connection you better use a very simple lightweight protocol like Aurora. PCIe is designed for very complex scenarios, for point-to-point this is most likely overkill.

Regards

Martin

venkat_vs2k2 · ‎11-19-2010

Hello Martin,

Thank you very much for your reply. Well eventually i have plans of scaling up the system. so i guessed PCIe would be appropriate. Could you tell me how this can be done ? is it just an isssue of decoding the PCIe TLP packets and generating packets to a corresponding Port number in the system ? Or is there some thing more complex like NTB that is involved ?

Thanks,

Venkat.

ckarman · ‎06-01-2011

Hi Venkat,

In PCIe system, all nodes are communicating to a single link partner (except if you're a switch/bridge). So, your FPGA is commiunicating to a single PCIe switch in this situation.

If you're an Endpoint, then it's not too much different than single Host <-> single EndPoint communication. And as you mentioned, you just have to give the right address when sending out requests or packets. Your flow control, DLLP handshake (ACK/NAK/etc) is only between the FPGA and the switch you're connected to. So if you look at it this way, you can see it's pretty much like node <-> node communication if you are an endpoint.

The host however will be quite different. The host will be in charge of enumerating (giving addresses) to each of the PCIe nodes and switches. The host will traverse through available buses and nodes and read their device ID. If the host see a valid device ID, then it will assign an address range for that device. The endpoint will look at this Configuration Write Type 0 from the host and it will decode the "completer ID" address field and it will know what address it's assigned to.

Same with the switches. It's in charge of knowing what nodes are below it (in the PCIe tree perspective) and route incoming packets correctly.

If you're in charge of desiging a host system and would like to know more about enumeration process, you can find books about this, one of them that I use is the PCIe system architecture from Mindshare.