12-08-2019 05:17 AM
In our custom board (K7325T-2, similiar as KC705), we implemented a 10G port using 10G Ethernet Subsystem IP.
When testing with IXIA (a network test equipment), we found that some packets are dropped when IXIA send packets that are not 8 bytes aligned at line rate.
For example, 64 bytes packet can achive full line rate while 65 bytes packet and 66 bytes packet can't achive line rate.
The testing setup is as follows:
IXIA -> 10GE port rx -> FIFO -> 10GE port tx -> IXIA
The refclk of this IP is 156.25mhz (sourcing by an osc directly) and IP's user side is axi-stream 64 bit wide.
By search in the forum, someone talk about DIC function, maybe i should enable that function?
what maybe the root cause of this problem?
any advice will be appreciated!
12-08-2019 10:49 PM
12-08-2019 10:49 PM
12-08-2019 11:27 PM
IMHO, DIC should probably be enabled by default.
IEEE802.3 does indicate that DIC is optional, so I guess that's why there's a control for it in the IP. However, there is no practical reason not to enable DIC. Every 10G switch and router out there will have it enabled. It might be better to remove that control from the GUI altogether.
12-08-2019 11:39 PM
12-10-2019 05:48 PM
We make products that can be configured to do that. It's needed to allow 100% thoughput whist also being able to deal with some bandwidth loss due to either clock tolerances or extra traffic being added.
A normal (i.e. standards compliant) 10G MAC running flat out will produce an average IFG of 12 and a minimum of 9. When we turn on our "extra bandwidth" feature, this drops to an average of 11 and a minimum of 8.
The loss of 4 bytes of IFG (producing a minimum IFG or 4) can come from an improperly designed clock rate adapter. These will insert or delete idles (groups of four bytes) from the IFG to match the difference in clock rates between link partners. The rules for deleting idles at 10G are listed in IEEE802.3 section 188.8.131.52.3.
This has a point "d" which states (in the 2005 version) "[w]hen deleting idles, the minimum IPG of five characters is maintained" and (in the 2008 version) "[t]he four characters following a Terminate control character are not deleted." This means that no device should ever output an IFG as short as four characters, and indicates that the Arista switch has a bug. [BTW, Our products don't have that bug.] Of course, that doesn't help your product interwork, leaving you no choice but to harden your design against such behaviour.
12-10-2019 05:50 PM
BTW, the bug could also have been from the clock rate compensation circuit in your product upstream of your MAC!
12-10-2019 06:16 PM
In my case, there was no clock rate compensation code; the MAC ran entirely in the RX clock domain and the clock domain transition took place in an AXI stream async FIFO after the MAC. But yes, I can see how that module could cause this either if written incorrectly or if the frequency difference was high enough, and I can also see how clock rate compensation could be tripped up by that if it came in over the wire. And I have the ILA trace of the issue; looking back at it I captured a terminate character in XGMII lane 2, 5 idles, and then a start character in XGMII lane 0. So it seems the IFG in this case was actually 6.
12-11-2019 08:46 PM
We put our clock compensation in the PCS and have the MAC in the system clock domain. One day when I have some free time (i.e. never) I'll implement both ways and see which works best. (For some of our customers, "best" = lowest latency.)
On the receive side, there's the added complication that the recovered Rx clock can stop (e.g. if it comes from a Xilinx transceiver). That's why I prefer to jump into the (reliable) system clock domain as early as possible.
12-16-2019 08:59 PM