cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted
Observer
Observer
658 Views
Registered: ‎02-05-2020

Ethernet through SFP28 Cage

Jump to solution

Looking to run Ethernet out of the 4 SFP28 ports on the RFSoC to push towards the 100 Gb/s they are capable of together. I am using the 100G ethernet core (cmac_usplus) and have had successful simulations and have met timing for my builds. My problem now is that we are having trouble finding something to interface with through the ports. Our project is supposed to consist of running 100GbE through the 4xSFP28 cage to a QSFP28 port on a NIC which then moves it to a GPU for further processing. We reached out to a manufacturer about a cable that has 4 SFP connectors on one end and a QSFP on the other end but they seemed to believe that the cable wouldn't work do to the connection being host to host and not involving a switch in between. Looking for any advice on the hardware we should target for what is receiving the data out of the FPGA in terms of cables, NICs, or any other tips. Also, with the ethernet out of the FPGA does the ip core encapsulate that data into an ethernet frame or do I need to create the frame myself? We figured that with only two systems connected we did not need to but thought the NIC may drop it if it doesn't have a legal header. Thanks in advance for any advice, help, or links. 

0 Kudos
1 Solution

Accepted Solutions
Highlighted
Explorer
Explorer
541 Views
Registered: ‎08-14-2013
100G Ethernet is tolerant of all sorts of bit muxing and what not, so there should be no issues making a 100G link with four SFP28. You can definitely get breakout cables that have a QSFP28 on one end and four SFP28 on the other end, both DACs and AOCs. Take a look at fiberstore as one possible supplier. If you use an AOC instead of a DAC, you'll also have to make sure you're driving all of the SFP28 control pins correctly - namely, txdisable has to be driven correctly otherwise you won't get a link. Yes, I have screwed that up several times before. Now, it seems like most NICs require RS-FEC to be enabled for 100G operation. So you'll likely have to enable that option on the CMAC core before you'll be able to bring up a 100G link. After that, it should be pretty much plug and play. I have successfully interfaced the CMAC core on VU3P and VU9P parts to a couple of different Mellanox NICs. The CMAC core does not add any headers, you'll have to do that yourself. And the CMAC does not properly handle short frames, so make sure you never try to send frames shorter than 60 bytes (64 min length frame minus 4 byte FCS). Otherwise they will get dropped and the CMAC could lock up. If you use the CMAC in AXI stream mode instead of LBUS, then you can use this module to pad the frames to 60 bytes: https://github.com/ucsdsysnet/corundum/blob/master/fpga/common/rtl/cmac_pad.v . Also, most NICs should be fully capable of receiving frames that don't have valid headers so long as you turn off MAC address filtering - the Ethernet header is just some addresses anyway, and they can have arbitrary values. It is probably a good idea to attach a valid header, though.

View solution in original post

4 Replies
Highlighted
Xilinx Employee
Xilinx Employee
570 Views
Registered: ‎05-01-2013

You don't need to send any data. IP core is able to send/receive IDLE and get the link up first.

So how about CMAC block_lock, synced and aligned signals? Does CMAC link up?

If not, you can try near end loopback test first.

Highlighted
Explorer
Explorer
542 Views
Registered: ‎08-14-2013
100G Ethernet is tolerant of all sorts of bit muxing and what not, so there should be no issues making a 100G link with four SFP28. You can definitely get breakout cables that have a QSFP28 on one end and four SFP28 on the other end, both DACs and AOCs. Take a look at fiberstore as one possible supplier. If you use an AOC instead of a DAC, you'll also have to make sure you're driving all of the SFP28 control pins correctly - namely, txdisable has to be driven correctly otherwise you won't get a link. Yes, I have screwed that up several times before. Now, it seems like most NICs require RS-FEC to be enabled for 100G operation. So you'll likely have to enable that option on the CMAC core before you'll be able to bring up a 100G link. After that, it should be pretty much plug and play. I have successfully interfaced the CMAC core on VU3P and VU9P parts to a couple of different Mellanox NICs. The CMAC core does not add any headers, you'll have to do that yourself. And the CMAC does not properly handle short frames, so make sure you never try to send frames shorter than 60 bytes (64 min length frame minus 4 byte FCS). Otherwise they will get dropped and the CMAC could lock up. If you use the CMAC in AXI stream mode instead of LBUS, then you can use this module to pad the frames to 60 bytes: https://github.com/ucsdsysnet/corundum/blob/master/fpga/common/rtl/cmac_pad.v . Also, most NICs should be fully capable of receiving frames that don't have valid headers so long as you turn off MAC address filtering - the Ethernet header is just some addresses anyway, and they can have arbitrary values. It is probably a good idea to attach a valid header, though.

View solution in original post

Highlighted
Observer
Observer
410 Views
Registered: ‎02-05-2020
I am using a DAC but still can't get the design to work. I have enabled RS-FEC for the 25G links but even my 10G and 1G designs fail which make me think that I am not driving the control pins correctly. The ZCU111 board only has a tx_disable pin outside of the gt pins and I have check all the jumpers. I have checked out your implementation to see if I could identify any issues going on with mine and saw that the ports sfp_mod and sfp_prs exist in yours and was wondering what these. Also, do I need to have auto-negotiation enable? In each test I have forced the NIC to turn auto-negotiation off. Also, I am trying to just set up the example design from Xilinx right now just to test out what I can do and the external loopback works but I can't connect to the NIC and don't know if there is anything more I need to do with the core to get it working.
0 Kudos
Highlighted
Explorer
Explorer
285 Views
Registered: ‎08-14-2013

If you have a DAC, then all of those extra pins (tx disable, rs, module detect), etc. are completely irrelevant.  All of the DACs that I have seen are totally passive and directly connect the main high speed data pins together, the only other part in there would be an I2C EEPROM to identify the cable. I don't have an AN license, so I have never used AN for 10G, 25G, or 100G. 

If the external loopback works, that makes me think maybe you're running at the wrong line rate or something along those lines.  Have you checked to make sure the reference oscillators are configured the way they need to be? 

0 Kudos