cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
TaekyungHeo
Contributor
Contributor
1,367 Views
Registered: ‎07-10-2020

Is 40GbE QSFP+ compatible with Alveo U50 (QSFP28)?

Jump to solution

I am trying to use the 100GbE QSFP28 module on Alveo U50 for my design.

Currently, I have "Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+" card and "40GbE QSFP cable".

I couldn't connect a 40GbE card to Alveo U50 as I have described in the other message.

Is it related to the compatibility between the cable (QSFP+) and Alveo U50 card (QSFP28)?

0 Kudos
Reply
1 Solution

Accepted Solutions
TaekyungHeo
Contributor
Contributor
1,306 Views
Registered: ‎07-10-2020

It turns out that there is no compatibility issue between them.

The key problem is the NIC that I used (Intel NIC does not work, and Mellanox's works).

Let me introduce my solution step-by-step

1. By following this answer record, I could bring up the IP core (rx_clk works).

However, I found that stat_rx_aligned is not asserted while rx_local_fault is asserted.

2. According to this post, disabling auto-negotiation is a solution to avoid rx_local_fault.

Unfortunately, I couldn't disable auto-negotiation of Intel NIC with ethtool.

3. Finally, I changed my NIC from Intel's to Mellanox's, and it works.

A researcher who wrote verilog-ethernet confirmed that using a Mellanox  ConnectX-5 is a good option to figure out working settings (link).

View solution in original post

0 Kudos
Reply
11 Replies
TaekyungHeo
Contributor
Contributor
1,307 Views
Registered: ‎07-10-2020

It turns out that there is no compatibility issue between them.

The key problem is the NIC that I used (Intel NIC does not work, and Mellanox's works).

Let me introduce my solution step-by-step

1. By following this answer record, I could bring up the IP core (rx_clk works).

However, I found that stat_rx_aligned is not asserted while rx_local_fault is asserted.

2. According to this post, disabling auto-negotiation is a solution to avoid rx_local_fault.

Unfortunately, I couldn't disable auto-negotiation of Intel NIC with ethtool.

3. Finally, I changed my NIC from Intel's to Mellanox's, and it works.

A researcher who wrote verilog-ethernet confirmed that using a Mellanox  ConnectX-5 is a good option to figure out working settings (link).

View solution in original post

0 Kudos
Reply
aforencich
Explorer
Explorer
1,210 Views
Registered: ‎08-14-2013

You should be able to use any optical module that you want in the U50 (and with any FPGA board, really).  Other devices can be artificially picky by baking module vendor ID checks into the firmware.

However, the U50 (and also the U280) do not currently provide any access to the module I2C interface, so you can run in to issues if the module contains a CDR and you're attempting to run at a non-standard line rate, such as attempting to use a QSFP28 module at 10G or 40G.  To make this work, you need to talk to the module over I2C and disable the module CDRs, but this is not currently possible on the U50 and U280. 

TaekyungHeo
Contributor
Contributor
1,174 Views
Registered: ‎07-10-2020

Thanks for sharing the issue, Alex!

I hope Xilinx lets us access and control the module over I2C also.

0 Kudos
Reply
GordonWu
Newbie
Newbie
814 Views
Registered: ‎12-04-2020

Hi! I also have a question about cable compatibility. Our team is going to build a testbed. There will be an EdgeCore switch connecting to an Alveo U250 card. And we want to use DAC cable(cheaper and enough cable length for our testbed). However, the sales rep warns us that this may cause compatibility issues and mentioned that the hardware might check vendor IDs. I noticed that you mentioned that Xilinx's FPGAs don't check this. Does this mean that if we buy a DAC cable that is EdgeCore-compatible, this cable should work just fine? Thank you in advance!

0 Kudos
Reply
aforencich
Explorer
Explorer
807 Views
Registered: ‎08-14-2013

Yes.  On the U250 (and VCU1525 and U200), the transceiver I2C interfaces are directly accessible from the FPGA.  Unless you implement the checking yourself, no checking is done.  In general, "compatibility" issues like that are simply whitelists in the firmware, with the intention of forcing customers to buy accessories from particular vendors.  Xilinx is not a vendor for DAC cables, QSFP transceivers, etc. so it is not in their interest to add this type of BS to their cards.  Many switch vendors, on the other hand, sell significantly marked-up cables and transceivers, and as such artificial "compatibility issues" directly help their bottom line.

Now, for the U50 and U280 the transceivers sit behind the BMC, and until Vivado 2020.2, the BMC firmware did not provide any access at all to the transceivers.  Now, in 2020.2, it seems like they added a very limited method for accessing a couple of transceiver registers that is probably all you need for 90% of what you might want to do.  But it's useless if you need to poke something outside of that small window, or even read something on a different page. 

sametcaglan
Observer
Observer
590 Views
Registered: ‎12-16-2015

Hi TaekyungHeo,

I am trying to communicate Mellanox NIC and KCU105 board via 40GbE QSFP+ cable, but stat_rx_aligned is down at FPGA side. NIC supports up to 100Gbps link speed.  Is there any need to Auto-Negotiation at FPGA side? Or should I close AN at NIC side?

Thanks.

0 Kudos
Reply
aforencich
Explorer
Explorer
571 Views
Registered: ‎08-14-2013

You don't need autonegotiation, but you do need to enable RS-FEC.  Mellanox NICs won't bring up a 100G link unless RS-FEC is enabled. 

sametcaglan
Observer
Observer
532 Views
Registered: ‎12-16-2015

Thanks Aforencich,

At FPGA side, I used 40G Ethernet Subsystem. For 40G link, is there also need to enable RS-FEC? For 40G Ethernet Subsytem, I enabled Clause 74 FEC but Clause 91 RS-FEC is not selectable. Is it required a licence?

0 Kudos
Reply
aforencich
Explorer
Explorer
514 Views
Registered: ‎08-14-2013

Not sure, I have never worked with 40G before, only 10G, 25G, and 100G.  It's possible that the Mellanox NICs require some form of FEC at 40G.  The 40G MAC is soft IP, I believe, so the MAC, AN, and RS-FEC I think are all separate licenses, vs. the 100G CMAC where the MAC and FEC are hard logic so only AN requires a separate license.

It's also possible that you're running in to some other issue that's preventing the link from working correctly - have you double-checked all of the transceiver sites and reference clock selections against the board?  Have you tried running the link at 10G?  Also, the KCU105 only has a pair of SFP+ cages which is not sufficient to run at 40G, so how are you connecting the QSFP+ cable to the board? 

0 Kudos
Reply
sametcaglan
Observer
Observer
510 Views
Registered: ‎12-16-2015

Thanks aforencich,

In fact, I use a custom board that has same FPGA as KCU105. For that reason, I said it as KCU105. I didn't pay any attention that KCU105 only has a pair of SFP+ cages which is not sufficient to run at 40G :). My board communicates with NI PXI Switch (supports up to 40G) over same 40GbE QSFP+ cable. On the other side, link goes down when connecting to Mellanox NIC with same cable.

0 Kudos
Reply
aforencich
Explorer
Explorer
506 Views
Registered: ‎08-14-2013

If you're not using a Xilinx dev board, then don't say you're using a Xilinx dev board just because you happen to be using the same FPGA that's on that particular dev board.  If you're using a dev board, folks on here have access to the schematics, have experience working with the board, know about potential board-specific considerations, etc.  However, if you rolled a custom board, then this is not the case and there are all sorts of other things that may need to be checked.  Saying you're using board X when you aren't can very well lead to a lot of wasted time due to bad assumptions.

Anyway, if it works with the PXI switch but not the Mellanox NIC, then that's potentially a good sign.  Does the mellanox NIC work with that switch?  Can you check what the NIC reports about the FEC configuration when it has a working 40G link? 

0 Kudos
Reply