100G CMAC Rx Sync Errors - Zync US +

[samlyn]

Hi

I'm struggling to set up two 100G CMAC PHY IPs, on a Zynq UltraScale+ RFSOC board. In short 'stat_rx_aligned' never asserts and 'stat_rx_synced_err' goes high across all lanes.

Device: xczu29dr-ffvf1760-2-e, Tool Version: 2019.2

I have connected the two lanes externally with a QSFP optical cable. Internally I have just looped the AXI4s interfaces, as I'm focusing on getting the correct rx alignment before checking data. 

• If I set gt_loopback to Near End PMA loopback, both lanes align correctly. In normal mode or with any Far End loopback I get the synced errors.
• Using 'gtwiz_reset_rx_datapath' doesn't help (synced error deasserts but reasserts when the reset is removed).
• The Auto-Negotiation option is turned off for both lanes

Both ref clocks are driven by the outputs of a Silicon Labs Si5341 chip, programmed over the I2C bus. I haven't verified the ref clocks directly, but a 3rd output connected to an SMA port & that showed the correct frequency on a scope. The ref clock outputs are set to 156.25MHz 2.5V LVDS.

I have attached a TCL script for generating my IP Integrator set up. It includes a VIO and IBERT for each PHY to assist in the debugging.

This looks like a hardware/cabling or clock issue to me but I'm running out of ideas on how to solve it. I've tried two separate optical cables & manufacturer. Any help or ideas would be greatly appreciated, Thanks

Additional info:

The CMAC 'init_clk' is a 100MHZ output from a Clock Manager IP, which is driven by a fixed 300MHZ input to the board, 'sys_clk'.

My XDC constraints file contains location & IO constaints for 'sys_clk' and the I2C pins. It also contains pin placement for the qsfp refclks. It is my understanding that all other necessary constraints are included in the IP generated XDC files; qsfp refclk IO standard, tranceiver pin location & IO by the CMAC IP, and 'create_clk' for 'sys_clk' and 'init_clk' handled by the CM IP.

My CMAC core bring up logic is as follows, derived from the UG and example design. This has worked for me on another US+ device (not Zync). init_clk is a 100MHz output from the Clk Manager IP and sys_reset <= not cm_locked 

rx_clk <= gt_txusrclk2

ctl_rx_en <= not usr_rx_rst;

ctrl_tx_send_rfi <= not stat_rx_aligned;

ctrl_tx_en <= stat_rx_aligned; 

I tried an IBERT eye scan between Tx: X012 (Lane 0 CMAC 0) and Rx: X0Y16(Lane 0 CMAC 1), and the eye was fully closed. I haven't used IBERT before but I'm assuming that was a correct link to set up.

[guozhenp]

1. Do you have 2 CMAC IP cores interconnected? If not, what's the link partner?

2. Can both CMAC link up in the near end loopback?

3. Do you have all the block_lock asserted?

4. Do both CMAC have the same CTL_TX_VL_LENGTH_MINUS1 and CTL_RX_VL_LENGTH_MINUS1 settings? Please confirm them carefully.

[samlyn]

1. Yes they are interconnected

2. Yes both look OK when set to Near end Loopback

3. stat_rx_block_lock asserts only in Near End Loopback. It remains deasserted otherwise.

4. Sorry how do I check these values? I try the TCL command 'report_property [get_ips cmac_usplus_0]' but the  CTL_TX_VL_LENGTH_MINUS1 and CTL_RX_VL_LENGTH_MINUS1 settings are not listed in the IP properties. I also can't see them in the IP settings window. I have not tried to manually set these so I assume they are still set to default

Thanks

[guozhenp]

These signals are IP core inputs. So you should be able to find them in the design source codes directly.

However, even block lock is not asserted in your case. It seems that the physical link doesn't work at all.

1. Please confirm the GT locations in your design (constraints in the XDC file). The GT pins are correct on your board.

2. You can try run IBERT first. Check if the link between these 2 boards are good.

[samlyn · Accepted solution]

Yes the GT locations were correct & the IBERT eye was fully closed, I checked the hardware/Physical layer and it turns out I wasn't disabling the Low Power Mode of the QSFPs correctly.

This solved the issue, thanks for the help