UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Observer juddwang2017
Observer
375 Views
Registered: ‎09-06-2018

10G Ethernet Subsystem packet loss when test under full line rate

Jump to solution

Hi ,

In our custom board (K7325T-2, similiar as KC705), we implemented a 10G port using 10G Ethernet Subsystem IP.

When testing with IXIA (a network test equipment), we found that some packets are dropped when IXIA send packets that are not 8 bytes aligned at line rate. 

For example, 64 bytes packet can achive full line rate while 65 bytes packet and 66 bytes packet can't achive line rate.

The testing setup is as follows:

IXIA -> 10GE port rx -> FIFO -> 10GE port tx -> IXIA

The refclk of this IP is 156.25mhz (sourcing by an osc directly) and IP's user side is axi-stream 64 bit wide.

 

More Info:

By search in the forum, someone talk about DIC function, maybe i should enable that function?

what maybe the root cause of this problem? 

any advice will be appreciated!

 

0 Kudos
1 Solution

Accepted Solutions
Participant aforencich
Participant
328 Views
Registered: ‎08-14-2013

Re: 10G Ethernet Subsystem packet loss when test under full line rate

Jump to solution
Yes, DIC must be enabled to hit full line rate at 10G. The problem is that 10G data is packed into 64 bit frames, and there are only 2 possible start offsets in each 64 bit frame. That means that the IFG can only be adjusted in increments of four bytes. With DIC disabled, that means frames that don't end on a 4 byte boundary results in IFGs that are all rounded up above 12 bytes, resulting in a slightly lower than expected rate. DIC enables 'stealing' up to 3 byte times from the IFG to maintain an average IFG of 12 byte times, enabling operation at the expected 10G line rate. IMHO, DIC should probably be enabled by default.

View solution in original post

0 Kudos
8 Replies
Participant aforencich
Participant
329 Views
Registered: ‎08-14-2013

Re: 10G Ethernet Subsystem packet loss when test under full line rate

Jump to solution
Yes, DIC must be enabled to hit full line rate at 10G. The problem is that 10G data is packed into 64 bit frames, and there are only 2 possible start offsets in each 64 bit frame. That means that the IFG can only be adjusted in increments of four bytes. With DIC disabled, that means frames that don't end on a 4 byte boundary results in IFGs that are all rounded up above 12 bytes, resulting in a slightly lower than expected rate. DIC enables 'stealing' up to 3 byte times from the IFG to maintain an average IFG of 12 byte times, enabling operation at the expected 10G line rate. IMHO, DIC should probably be enabled by default.

View solution in original post

0 Kudos
318 Views
Registered: ‎01-08-2012

Re: 10G Ethernet Subsystem packet loss when test under full line rate

Jump to solution

@aforencich wrote:
IMHO, DIC should probably be enabled by default.

IEEE802.3 does indicate that DIC is optional, so I guess that's why there's a control for it in the IP.  However, there is no practical reason not to enable DIC.  Every 10G switch and router out there will have it enabled.  It might be better to remove that control from the GUI altogether.

0 Kudos
Participant aforencich
Participant
313 Views
Registered: ‎08-14-2013

Re: 10G Ethernet Subsystem packet loss when test under full line rate

Jump to solution
Actually, some devices will cut down the IFG by a lot more than the spec ostensibly allows for, even with DIC enabled. I was getting dropped packets with an Arista 40G switch (running at 10G) that I traced to the switch occasionally producing an IFG of 4 byte times, and my 10G MAC was not handling that correctly (didn't detect the start control character) and dropped the frame as a result. Had to make some adjustments to the MAC so it would handle that properly.
0 Kudos
264 Views
Registered: ‎01-08-2012

Re: 10G Ethernet Subsystem packet loss when test under full line rate

Jump to solution

We make products that can be configured to do that.  It's needed to allow 100% thoughput whist also being able to deal with some bandwidth loss due to either clock tolerances or extra traffic being added.

A normal (i.e. standards compliant) 10G MAC running flat out will produce an average IFG of 12 and a minimum of 9.  When we turn on our "extra bandwidth" feature, this drops to an average of 11 and a minimum of 8.

The loss of 4 bytes of IFG (producing a minimum IFG or 4) can come from an improperly designed clock rate adapter.  These will insert or delete idles (groups of four bytes) from the IFG to match the difference in clock rates between link partners.  The rules for deleting idles at 10G are listed in IEEE802.3 section 48.2.4.2.3.

This has a point "d" which states (in the 2005 version) "[w]hen deleting idles, the minimum IPG of five characters is maintained" and (in the 2008 version) "[t]he four characters following a Terminate control character are not deleted."  This means that no device should ever output an IFG as short as four characters, and indicates that the Arista switch has a bug.  [BTW, Our products don't have that bug.]  Of course, that doesn't help your product interwork, leaving you no choice but to harden your design against such behaviour.

0 Kudos
263 Views
Registered: ‎01-08-2012

Re: 10G Ethernet Subsystem packet loss when test under full line rate

Jump to solution

BTW, the bug could also have been from the clock rate compensation circuit in your product upstream of your MAC!

0 Kudos
Participant aforencich
Participant
254 Views
Registered: ‎08-14-2013

Re: 10G Ethernet Subsystem packet loss when test under full line rate

Jump to solution

In my case, there was no clock rate compensation code; the MAC ran entirely in the RX clock domain and the clock domain transition took place in an AXI stream async FIFO after the MAC.  But yes, I can see how that module could cause this either if written incorrectly or if the frequency difference was high enough, and I can also see how clock rate compensation could be tripped up by that if it came in over the wire.  And I have the ILA trace of the issue; looking back at it I captured a terminate character in XGMII lane 2, 5 idles, and then a start character in XGMII lane 0.  So it seems the IFG in this case was actually 6. 

0 Kudos
212 Views
Registered: ‎01-08-2012

Re: 10G Ethernet Subsystem packet loss when test under full line rate

Jump to solution

We put our clock compensation in the PCS and have the MAC in the system clock domain.  One day when I have some free time (i.e. never) I'll implement both ways and see which works best.  (For some of our customers, "best" = lowest latency.)

On the receive side, there's the added complication that the recovered Rx clock can stop (e.g. if it comes from a Xilinx transceiver).  That's why I prefer to jump into the (reliable) system clock domain as early as possible.

0 Kudos
Participant aforencich
Participant
136 Views
Registered: ‎08-14-2013

Re: 10G Ethernet Subsystem packet loss when test under full line rate

Jump to solution
Seems to me that a standard async FIFO would have less latency than an elastic buffer with a depth of 16 or 32 words. But your point about the clock stopping is a good one.
0 Kudos