UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Explorer
Explorer
839 Views
Registered: ‎05-14-2017

100G ethernet configuration

Based on PG203 document v3.0 when configuration the IP in the General TAB.

The GT Location Selection has either:

1) Include GT subcore in core and

2) Include GT subcore in example design

Is selectionn 1 For a real design and selection 2 to use in only in the example for evaluation?

0 Kudos
22 Replies
Xilinx Employee
Xilinx Employee
817 Views
Registered: ‎09-05-2018

Re: 100G ethernet configuration

Hey @tchin123,

I see people use option 2 when configuring a second Ethernet IP which will share the same GT's.

Besides that, I think option 1 is easier to use. You can use either core for evaluation.

Nicholas Moellers

Xilinx Worldwide Technical Support
Explorer
Explorer
768 Views
Registered: ‎05-14-2017

Re: 100G ethernet configuration

In my design, I'm interfacing the 100G Ethernet IP (PG203) with the ERNIC IP (PG332).

I beleive from the IP Wizard I should :

1) select the Ethernet Option-1 (Include GT subcore in core) from the IP General Tab and

2) select the AXI4 from the USE Interface Tab

3) then remove the CMAC_USPLUS_0_PKT_GEN_MON sub-block and connect the CMAC_USPLUS_0.v it to the ERNIC IP Port?

Would this be the corect initial setup to my FPGA design?

0 Kudos
Xilinx Employee
Xilinx Employee
754 Views
Registered: ‎09-05-2018

Re: 100G ethernet configuration

Hey @tchin123 ,

That seems like a reasonable plan. Check page 41 for design guidelines which you may find helpful. You could also reference the hardware reference design for the ERNIC which you should have access to.

Nicholas Moellers

Xilinx Worldwide Technical Support
0 Kudos
Explorer
Explorer
739 Views
Registered: ‎05-14-2017

Re: 100G ethernet configuration

Hi, you mention page 40, what is on page 40,;at PG203, v3.0 it shows attribute description only?

I'm also looking for the CMAC interface 512-bit format. Since I'm planning to select the AXI4 user interface during IP wizard configuration, I might need to know what is being received and transmitted on this interface in order to decode and process the packet in my logic application block.

I believe the Ethernet header and FCS frame is handle by the CMAC IP and then stripped away and the remaining Infiniband Transport (RoCE v2) , TCP/IP and application data reside on this bus? Do I need to handle the processing of this interface format (LBUS oir AXI4) or is it ransparent to the user because my next IP (ERNIC IP) would process it and eventually will only output the application data to the user?

Either way, shouldn't the 100B Ethernet CMAC IP interface (LBUS / AXI4) be describe somewhere in the PG203 document if the user want to process this bus manually, in respect to what packet field or data format is actually presented here sequentially at this interface or am I missing some concept here?

 

0 Kudos
Adventurer
Adventurer
675 Views
Registered: ‎07-27-2018

Re: 100G ethernet configuration

Hi @tchin123 

regarding following question:

Either way, shouldn't the 100B Ethernet CMAC IP interface (LBUS / AXI4) be describe somewhere in the PG203 document if the user want to process this bus manually, in respect to what packet field or data format is actually presented here sequentially at this interface or am I missing some concept here?

I think section User Side LBUS Interface (pag 58 of PG203 v3.0), can help you to understand the LBUS rx/tx functioning, in particular the timming diagram show you how to toggle the SOP, EOP and the other necessary signals.

For AXI4 part you can see Core Bring Up Sequence (pag. 206).

Basically you need a control block to initialize the CMAC and then you can proceed to read an write on AXI Stream ports.

 

Hope this can help.

Regards

 

 

 

Explorer
Explorer
661 Views
Registered: ‎05-14-2017

Re: 100G ethernet configuration

I understand the LBUS / AXI4 timing and protocol but without knowing the data field on this bus, the user cannot process them.

For example, what is display here, the Ethernet frame, the payload, the length field and so on.Not knowing what is transmitted on the bus itself and in what order, the user cannot process them correctly.

In other word, what does the IP present on this bus and in what order

0 Kudos
Adventurer
Adventurer
646 Views
Registered: ‎07-27-2018

Re: 100G ethernet configuration

Hi @tchin123 

always from pg203

"The transmitter accepts packet-oriented data, packages the data in accordance with the
IEEE 802.3 Specification, and sends that packaged data to the serial transceiver interface.
The transmitter has control/configuration inputs to shape the data packaging to meet
design-specific requirements."

Build an ethernet packet!

Run the simulation of the example design  provided with the ip and watch the data on the lbus, this can help you to understand...

 

Ps 512 bit LBUS is suitable to send a minimum eth packet of 64byte in one shot....

 

 

Xilinx Employee
Xilinx Employee
556 Views
Registered: ‎05-01-2013

回复: 100G ethernet configuration

Option2 moves some shared logic from IP internal outside with sources available

So they're the same after you generate the IP core example designs

0 Kudos
Explorer
Explorer
538 Views
Registered: ‎05-14-2017

Re: 100G ethernet configuration

OK, according to your explanation, it seem like the LBUS or AXI4 512-bit bus, during receiver mode will have the complete IEEE 802.3 frame on this bus.

Therefore all the 22-bytes of the Ethernet L2 header + payload + FCS(4-byte) will be received by the user?

This seem strange because shouldn't the 100B Ethernet IP already process the MAC source and destination address, as well as the FCS first before passing the rest of the frame (payload)? Why should it put this information back on the LBUS/AXI4 bus?

If all the field of the IEEE 802 is presented on this bus then doe this mean the user has to process the ethernet header and compute and check the CRC FCS field?

0 Kudos
Adventurer
Adventurer
479 Views
Registered: ‎07-27-2018

Re: 100G ethernet configuration

Hi @tchin123  following my considerations:

 

Therefore all the 22-bytes of the Ethernet L2 header + payload + FCS(4-byte) will be received by the user?

- Yes you are managing a "Layer 2 Ethernet frame" so an Ethernet MAC frame, and on the LBUS you will read all frame

This seem strange because shouldn't the 100B Ethernet IP already process the MAC source and destination address, as well as the FCS first before passing the rest of the frame (payload)? Why should it put this information back on the LBUS/AXI4 bus?

If all the field of the IEEE 802 is presented on this bus then doe this mean the user has to process the ethernet header and compute and check the CRC FCS field?

You can just modify the FCS presentation both in TX and RX by mean of signals like:

CTL_RX_DELETE_FCS, CTL_RX_IGNORE_FCS, CTL_TX_FCS_INS_ENABLE, CTL_TX_IGNORE_FCS

Please read section "Attribute Descriptions" on page 38, the description will clarify you thiese aspects.

Reagards.

0 Kudos
Explorer
Explorer
430 Views
Registered: ‎05-14-2017

Re: 100G ethernet configuration

See if I get this right:

The 100B ethernet IP core will put the complete 802.3 frame on the RX LBUS (512-bit) when RX_SOP=H. This will include the DA, SA, Length/Type,Payload and FCS. Please confirm.

Then during transmit, when TX_SOP=H, the user will place 802.3 Ethernet frame on the TX LBUS which is also: DA, SA, Length/Type,Payload and FCS?

Now the tricky part, if user set ctl_tx_fcs_ins_enable=H

then we instruct the IP core to add FCS. Now does the user still place the dummy FCS (4-byte) on the LBUS or does the user shrink the original 802.3 frame by 4-byte which mean the frame will end right after the payload?

 

0 Kudos
Observer aforencich
Observer
410 Views
Registered: ‎08-14-2013

Re: 100G ethernet configuration

In the default configuration, the AXI stream interface carries the Ethernet header and payload only.  So 14 bytes of header plus the payload.  The MAC computes and inserts the FCS on TX, and checks and strips the FCS on receive.  I think you can change some of this in the configuration options if you need the FCS for some reason.  Note that unlike other MACs, the CMAC does not pad transmit frames of less than 64 bytes, like many other MACs do.  You'll have to do that yourself, which is trivial for a 512 bit interface because the whole interface is 64 bytes wide to begin with. 

0 Kudos
Explorer
Explorer
374 Views
Registered: ‎05-14-2017

Re: 100G ethernet configuration

Thanks, That make it very clear as to what to expect on the 512-bit LBUS/AXI4 bus.

This should be included in the documentation PG203 since this information is very important because the user need to know how to process the field during receive and need to format and structure the bus during transmit. 

During Rx/Tx, when SOP=H, is the Dest Addr received or transmitted on upper byte or lower byte of the 512-bit bus?

If there are layer 3 fields like TCP/IP then, these fields should follow right after the Ether Lenght/Type 802.3 frame and presented on the 512-bit bus?

An example would solidify this interface for the user.

 

0 Kudos
Observer aforencich
Observer
356 Views
Registered: ‎08-14-2013

Re: 100G ethernet configuration

With the AXI interface, the first byte of the packet is [7:0], the next is [15:8], etc.  So the dest MAC address will cover [7:0]...[47:40], the source MAC covers [55:48]...[95:78], etc..  IIRC the LBUS interface is byte-reversed, with the first byte of the packet in [511:504]. 

0 Kudos
Explorer
Explorer
347 Views
Registered: ‎05-14-2017

Re: 100G ethernet configuration

I wasn't aware that Bye ordering between LBUS and AXI4 bus is reversed. This is very important, since the I'm planning to use the AXI4 interface instead of the LBUS.. Thanks for pointing this out. 

The PG203 document should really should cover this.Appreciate the input.

I don't know why, the LBUS/AXI4 data bus field structure was never cover, how does anyone plan to use this anyway

 

0 Kudos
Observer aforencich
Observer
338 Views
Registered: ‎08-14-2013

Re: 100G ethernet configuration

It is mentioned in a number of places that the first byte is on bits [127:120] for the LBUS interface.  But the AXI stream description is literally two sentences.  I suppose the assumption is that it works the same as the 10G MACs (and those have more detailed descriptions), just with more bits, but this isn't detailed in the CMAC documentation anywhere. 

0 Kudos
Explorer
Explorer
287 Views
Registered: ‎05-14-2017

Re: 100G ethernet configuration

The LBUS interface for bus format and endian is covered in the documention but in term of using the AXI4 interface, one have to access the 10G documentation.

Since as a first user of the 100G_MAC I was focus on PG203 only. Thanks for pointing out the other documentation. Who would even know to reference the 10G unless the user has use it previously. 

Since the newer 100G MAC (2019) has the AXI4 option, I would assume it is best to use this instead of the LBUS. Even though they are both 512-bit bus, the AXI4 bus has a simplier interface because the user do not have to process and calculate the "TX_MTYIN / RX_MTYIN" signals that exist on the LBUS in each of the 128-bit segment.

The AXI4 only has the TLAST signal instead which cover the comjplete 512-bit interface. and since it is little endian, the TLAST signal is much simplier.

any thought

 

0 Kudos
Observer aforencich
Observer
266 Views
Registered: ‎08-14-2013

Re: 100G ethernet configuration

So I actually did some thinking about why they have the LBUS interface vs. an AXI interface.  Taking into consideration the min frame length and standard interframe gap, you can saturate a 100 Gbps link with a 512 bit AXI interface running at about 280 MHz with the "worst case" frame length (65 bytes).  With four segments as in LBUS, you can saturate the link at closer to 200 MHz.  But with the whole core running at 322 MHz, AXI is more than sufficient and the LBUS interface doesn't make much sense.  My thinking is that the core was designed to be asynchronous, in which case being able to feed it with a segmented interface at a lower clock speed makes a lot of sense.  However, perhaps it didn't work in silicon the way it was intended, so they just changed the documentation to require the whole core to run in the TX clock domain. 

0 Kudos
Highlighted
Explorer
Explorer
192 Views
Registered: ‎05-14-2017

Re: 100G ethernet configuration

So the LBUS and the AXI4 actuallly runs at different  rate, I didn't even think that deeply. 

I was more thinking at the functional side where the LBUS actually allow back to back frame within the same "CYCLE" where the start of SOP of frame_2 can actually start immediately right after the EOP of frame_1 in the same cycle.

Being so, the calculation of the LBUS signal, MTYOUT for each segment during EOP cycle is more involve in determining its value

Where as in the AXI4 bus, when the frame ends, the next frame doesn't start until the next cycle and the AXI4 signal TLAST is determine more straight forward. .

So since back to back frame in AXI4 bus doesn't occur in the same cycle, make it more easier to use in my opinion, does that make sense?

0 Kudos
Observer aforencich
Observer
174 Views
Registered: ‎08-14-2013

Re: 100G ethernet configuration

Exactly - AXI is simpler, easier to use, but has a high overhead due to empty byte-lanes at the end of a frame, while LBUS can utilize some of those empty byte lanes at the cost of being a gigantic pain in the rear. 

0 Kudos
Explorer
Explorer
161 Views
Registered: ‎05-14-2017

Re: 100G ethernet configuration

Thanks for all your input.

I was wondering, what higher level Xilinx IP (layer 3) have you use to connect to the 100G MAC IP.

I have been inquiring the Xilinx ERNIC IP at another forum for a while and greceive no reply from anyone and was hoping maybe I could pass some of my question in your direction?

0 Kudos
Observer aforencich
Observer
116 Views
Registered: ‎08-14-2013

Re: 100G ethernet configuration

I am not using any Xilinx IP aside from the PCIe hard IP core and the CMAC core, the rest of it is 100% custom, and 100% open source: https://github.com/ucsdsysnet/corundum .
0 Kudos