UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Visitor svarun
Visitor
1,797 Views
Registered: ‎09-06-2017

Virtex Ultrascale 100G Ethernet CMAC IP - Bit Errors in Transmission

Jump to solution

I'm using XCVU095 on an Alpha Data ADM-PCIE-8V3 board. I'm trying to benchmark the 100G Ethernet CMAC IP in loopback but I'm seeing bit errors in transmission occasionally.

 

I'm using the Xilinx AXI Traffic Generator to generate and validate the received data. What I observe with ILAs is that after some random number of transactions (I've seen 1,000,000+ error-free), one of the RX LBUS 128 bit segments has some bit errors. I see the correct information a few cycles before on the TX LBUS side but on the RX side, I usually (if not always) see three non-consecutive bits flipped.

 

The CMAC IP is in CAUI-4 mode, using most of the default options. I tried enabling pipeline registers but that didn't fix the issue. I haven't tried RS-FEC yet but I wanted to first get a sense of my options in terms of changes I could make. Using the IBERT IP, I did see zero-error rate transmission so I know it should be achievable. I also saw this thread regarding TXPRECURSOR, TXPOSTCURSOR and TXDIFFCTRL which might be applicable but I don't see these options in the IP GUI.

0 Kudos
1 Solution

Accepted Solutions
Scholar austin
Scholar
2,566 Views
Registered: ‎02-27-2008

Re: Virtex Ultrascale 100G Ethernet CMAC IP - Bit Errors in Transmission

Jump to solution

Meeting timing by 61ps is too tight,

 

The default for set_system_jitter is 100 ps, and often system jitter is more like 300 ps.  So id you do not know your system jitter, set it larger (with TCL command set_system_jitter = 300 ps), and then see if you meet timing.

 

This is most likely your problem:  you have not taken all jitter sources into account, and timing occasionally fails.

Austin Lesea
Principal Engineer
Xilinx San Jose
0 Kudos
7 Replies
Scholar austin
Scholar
1,772 Views
Registered: ‎02-27-2008

Re: Virtex Ultrascale 100G Ethernet CMAC IP - Bit Errors in Transmission

Jump to solution

sv,

 

Does you design meet timing (has sufficient slack)?  If you have insufficient slack, you will occasionally get a burst of errors.  You may also be getting errors in your environment from noise or ripple on your power rails.  What is your system jitter?  What is it constrained tO/  Have you measured it and verified it is as constrained?  Most common cause is you switch too many IO signals (or even internal signals) on the same clock edge causing ground and Vcc bounce.  FEC is useful only if you are sure you have the best response you can get -- using FEC when you haven't found all the causes of errors means another board (a different device) may be completely broken as the FEC is incapable of fixing everything.

Austin Lesea
Principal Engineer
Xilinx San Jose
0 Kudos
Visitor svarun
Visitor
1,760 Views
Registered: ‎09-06-2017

Re: Virtex Ultrascale 100G Ethernet CMAC IP - Bit Errors in Transmission

Jump to solution

Hi Austin,

 

The design does meet timing (worst negative slack is 0.061ns). I could be missing constraints though since I've only defined the clocks and bitstream configuration information.

 

As for system jitter, I'm not sure. I haven't constrained the design to jitter. I saw that you mentioned here that I could measure it by routing a clock out to DDR DFF IOB pin and measure jitter on that. In this case, would the relevant clock be the GT reference clock? If so, I know on the board that the clock is generated using the SI5328 IC which supposedly has "sub 1ps jitter performance".

 

There's also the Auto Negotiation/Link Training option which I haven't enabled. Could this help?

 

Other information that might be relevant: I'm using Vivado 2017.2 on Linux. I'm using an external loopback cable to route the data back to the same QSFP port. I'll also try using internal loopback in the meantime.

 

Thanks for your help.

0 Kudos
Scholar austin
Scholar
2,567 Views
Registered: ‎02-27-2008

Re: Virtex Ultrascale 100G Ethernet CMAC IP - Bit Errors in Transmission

Jump to solution

Meeting timing by 61ps is too tight,

 

The default for set_system_jitter is 100 ps, and often system jitter is more like 300 ps.  So id you do not know your system jitter, set it larger (with TCL command set_system_jitter = 300 ps), and then see if you meet timing.

 

This is most likely your problem:  you have not taken all jitter sources into account, and timing occasionally fails.

Austin Lesea
Principal Engineer
Xilinx San Jose
0 Kudos
Visitor svarun
Visitor
1,735 Views
Registered: ‎09-06-2017

Re: Virtex Ultrascale 100G Ethernet CMAC IP - Bit Errors in Transmission

Jump to solution

Hi Austin,

 

Thanks for the tip. I'll try to increase the slack.

 

I did some more testing and found that both the near-end PCS and PMA loopback modes exhibited zero errors. I tried the far end loopback modes but there seemed to be no transactions occurring. Is this still indicative of a timing issue? The timing margin is about the same in all cases.

0 Kudos
1,710 Views
Registered: ‎01-08-2012

Re: Virtex Ultrascale 100G Ethernet CMAC IP - Bit Errors in Transmission

Jump to solution

@svarun wrote:
I also saw this thread regarding TXPRECURSOR, TXPOSTCURSOR and TXDIFFCTRL which might be applicable but I don't see these options in the IP GUI.

Try "Enable Additional GT Control and Status Ports" on the "CMAC/GT Selection and Configuration" tab in the CMAC GUI.

 

That you get 0 errors with PMA loopback says it's probably not a problem inside the FPGA fabric.  The most likely causes are: PCB loss (which can be ameliorated with the Tx EQ controls, as you surmised); some other signal integrity issue; power supply noise; clock jitter.

 

I use Si5340 to generate 300 to 600MHz gtrefclk on my boards but the Si5328 you are using should be more than good enough.

 

Another thing to consider: the GTY DFE is perhaps not as good as you might expect it to be.  I have yet to observe a condition where the DFE gives a better looking eye than the LPM Rx equaliser.  I have also observed conditions when the DFE seemed unable to converge on a 0 error setting.  It would mostly come good after a reset (or two).

A lot of issues went away after I switched to LPM.

 

Allan

0 Kudos
Visitor svarun
Visitor
1,681 Views
Registered: ‎09-06-2017

Re: Virtex Ultrascale 100G Ethernet CMAC IP - Bit Errors in Transmission

Jump to solution

An update for posterity:

 

I looked at IBERT again and copied the Tx EQ settings from there and the insertion loss from the IP instantiation itself that gave me error free transmission. However, using these settings in the CMAC didn't work (still bit errors). In fact, the higher insertion loss number made false rxena signals appear.

 

I addressed the close margin of timing as Austin suggested and raised the worst case slack to ~300 ps from ~60ps. This design, using the default values for the Tx EQ, gave me error free transmissions in at least 1E9 transactions. These values are using the default system jitter setting.

 

As a sanity check, I increased the system jitter setting to 300ps and tried my original design and the one with improved slack. The original failed timing by ~500ps. the new one still passed timing albeit by ~60ps.

 

It's safe to say that incorrect timing constraints gave me the issue. Thank you to both Austin and Allan. I was about to do a massive sweep of the Tx EQ parameters which I luckily don't need to.

0 Kudos
Scholar austin
Scholar
1,676 Views
Registered: ‎02-27-2008

Re: Virtex Ultrascale 100G Ethernet CMAC IP - Bit Errors in Transmission

Jump to solution

Glad you narrowed it down,

 

Austin Lesea
Principal Engineer
Xilinx San Jose
0 Kudos