cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
bkuschak
Adventurer
Adventurer
2,147 Views
Registered: ‎10-01-2013

GTH channel bonding not working

Jump to solution

I'm trying to get channel bonding to work with an 8 lane 8b/10b device.

 

I configured GTWizard per "Using RX Channel Bonding" on p283 of UG576 (v1.5). I used the method described in https://www.xilinx.com/support/answers/62140.html for choosing the master/slave daisy chaining.

 

  • KCU105 board using Quads 227/228. 32-byte external / 40-byte internal data width.

  • My CB sequence is: 0xBC(K=1) 0x60(K=0) 0x60(K=0) 0x60(K=0). Distance between these codes is at least 69 userclks.

  • CB is configured for 1 sequence of length 4. Max channel bonding level = 3. Don't cares unselected, inverted disparity unselected. One K-char.
  • CHAN_BOND_MAX_SKEW is set larger (tried 7 and 14) than the amount of skew that I observe (0 to 6 RXUSERCLK2 cycles).
  • No clock correction. Same clock used for source device and MGTREFCLK.

I verified my periodic CB sequence is being detected by the CB logic, by changing its value at the source and noting the effect on the FPGA. I see rxchanbondseq_out, rxchanisaligned_out, and rxchanrealign_out are being asserted (as displayed in a VIO). Looking at the output data from the GT, however, the data is not deskewed. Each time I reinitialize my source device the lane-to-lane skew is different, and it is not stable. The rxchanisaligned_out does seem to reflect correctly which lanes are currently aligned.

 

Any ideas what I'm doing wrong? Here's the relevant snippet of code:

 

wire [C_NUM_LANES-1:0] gt_rx_chbonden_in;

wire [C_NUM_LANES-1:0] gt_rx_chbondmaster_in;
wire [C_NUM_LANES-1:0] gt_rx_chbondslave_in;
wire [5*C_NUM_LANES-1:0] gt_rx_chbondi_in, gt_rx_chbondo_out;
wire [3*C_NUM_LANES-1:0] gt_rx_chbondlevel_in;
wire [C_NUM_LANES-1:0] gt_rx_chanisaligned_out;
wire [C_NUM_LANES-1:0] gt_rx_chanrealign_out;
wire [C_NUM_LANES-1:0] gt_rx_chanbondseq_out;

 

// We will use LANE[4] as the channel bonding master. The recommendation is to use the lane closest to the middle, to better meet timing.
assign gt_rx_chbondmaster_in[0] = 1'b0;
assign gt_rx_chbondmaster_in[1] = 1'b0;
assign gt_rx_chbondmaster_in[2] = 1'b0;
assign gt_rx_chbondmaster_in[3] = 1'b0;
assign gt_rx_chbondmaster_in[4] = 1'b1; // master
assign gt_rx_chbondmaster_in[5] = 1'b0;
assign gt_rx_chbondmaster_in[6] = 1'b0;
assign gt_rx_chbondmaster_in[7] = 1'b0;
assign gt_rx_chbondslave_in[0] = 1'b1;
assign gt_rx_chbondslave_in[1] = 1'b1;
assign gt_rx_chbondslave_in[2] = 1'b1;
assign gt_rx_chbondslave_in[3] = 1'b1;
assign gt_rx_chbondslave_in[4] = 1'b0; // not slave
assign gt_rx_chbondslave_in[5] = 1'b1;
assign gt_rx_chbondslave_in[6] = 1'b1;
assign gt_rx_chbondslave_in[7] = 1'b1;

 

// To handle 8 channels we need to split the chain twice.
// Refer to Issue 4 here: https://www.xilinx.com/support/answers/62140.html
assign gt_rx_chbondlevel_in[0*3 +: 3] = 3'd0; // Lane[0] slave = level 0
assign gt_rx_chbondlevel_in[1*3 +: 3] = 3'd0; // Lane[1] slave = level 0
assign gt_rx_chbondlevel_in[2*3 +: 3] = 3'd1; // Lane[2] slave = level 1
assign gt_rx_chbondlevel_in[3*3 +: 3] = 3'd2; // Lane[3] slave = level 2
assign gt_rx_chbondlevel_in[4*3 +: 3] = 3'd3; // Lane[4] master = level 3
assign gt_rx_chbondlevel_in[5*3 +: 3] = 3'd2; // Lane[5] slave = level 2
assign gt_rx_chbondlevel_in[6*3 +: 3] = 3'd1; // Lane[6] slave = level 1
assign gt_rx_chbondlevel_in[7*3 +: 3] = 3'd0; // Lane[7] slave = level 0


// Daisy chain channel bonding ports from master to slaves to implement the 3 chains
assign gt_rx_chbondi_in[0*5 +: 5] = gt_rx_chbondo_out[2*5 +: 5]; // 2 goes to both 1 and 0
assign gt_rx_chbondi_in[1*5 +: 5] = gt_rx_chbondo_out[2*5 +: 5]; // 2 goes to both 1 and 0
assign gt_rx_chbondi_in[2*5 +: 5] = gt_rx_chbondo_out[3*5 +: 5]; // 3 goes to 2
assign gt_rx_chbondi_in[3*5 +: 5] = gt_rx_chbondo_out[4*5 +: 5]; // master goes to 3 and 5
assign gt_rx_chbondi_in[4*5 +: 5] = 5'd0; // master - no input
assign gt_rx_chbondi_in[5*5 +: 5] = gt_rx_chbondo_out[4*5 +: 5]; // master goes to 3 and 5
assign gt_rx_chbondi_in[6*5 +: 5] = gt_rx_chbondo_out[5*5 +: 5]; // 5 goes to 6
assign gt_rx_chbondi_in[7*5 +: 5] = gt_rx_chbondo_out[6*5 +: 5]; // 6 goes to 7
assign gt_rx_chbonden_in = ~0;

 

// Xilinx GTWizard for 8 lanes.
gtwizard_ultrascale_0 gt (
  .gtwiz_userclk_tx_reset_in (0), // input wire [0 : 0] gtwiz_userclk_tx_reset_in
  .gtwiz_userclk_tx_srcclk_out (), // output wire [0 : 0] gtwiz_userclk_tx_srcclk_out
  .gtwiz_userclk_tx_usrclk_out (), // output wire [0 : 0] gtwiz_userclk_tx_usrclk_out
  .gtwiz_userclk_tx_usrclk2_out (), // output wire [0 : 0] gtwiz_userclk_tx_usrclk2_out
  .gtwiz_userclk_tx_active_out (), // output wire [0 : 0] gtwiz_userclk_tx_active_out
  .gtwiz_userclk_rx_reset_in (0), // input wire [0 : 0] gtwiz_userclk_rx_reset_in
  .gtwiz_userclk_rx_srcclk_out (), // output wire [0 : 0] gtwiz_userclk_rx_srcclk_out
  .gtwiz_userclk_rx_usrclk_out (), // output wire [0 : 0] gtwiz_userclk_rx_usrclk_out
  .gtwiz_userclk_rx_usrclk2_out (gt_rx_usrclk2_out), // output wire [0 : 0] gtwiz_userclk_rx_usrclk2_out
  .gtwiz_userclk_rx_active_out (), // output wire [0 : 0] gtwiz_userclk_rx_active_out
  .gtwiz_reset_clk_freerun_in (clk_freerunning), // input wire [0 : 0] gtwiz_reset_clk_freerun_in
  .gtwiz_reset_all_in (0), // input wire [0 : 0] gtwiz_reset_all_in
  .gtwiz_reset_tx_pll_and_datapath_in (0), // input wire [0 : 0] gtwiz_reset_tx_pll_and_datapath_in
  .gtwiz_reset_tx_datapath_in (0), // input wire [0 : 0] gtwiz_reset_tx_datapath_in
  .gtwiz_reset_rx_pll_and_datapath_in (0), // input wire [0 : 0] gtwiz_reset_rx_pll_and_datapath_in
  .gtwiz_reset_rx_datapath_in (0), // input wire [0 : 0] gtwiz_reset_rx_datapath_in
  .gtwiz_reset_rx_cdr_stable_out (), // output wire [0 : 0] gtwiz_reset_rx_cdr_stable_out
  .gtwiz_reset_tx_done_out (), // output wire [0 : 0] gtwiz_reset_tx_done_out
  .gtwiz_reset_rx_done_out (gt_rx_reset_done), // output wire [0 : 0] gtwiz_reset_rx_done_out
  .gtwiz_userdata_tx_in (0), // input wire [255 : 0] gtwiz_userdata_tx_in
  .gtwiz_userdata_rx_out (gt_rxd), // output wire [255 : 0] gtwiz_userdata_rx_out
  .gtrefclk01_in ({rx_refclk, rx_refclk}), // input wire [1 : 0] gtrefclk01_in
  .qpll1outclk_out (), // output wire [1 : 0] qpll1outclk_out
  .qpll1outrefclk_out (), // output wire [1 : 0] qpll1outrefclk_out
  .gthrxn_in (rxn), // input wire [7 : 0] gthrxn_in
  .gthrxp_in (rxp), // input wire [7 : 0] gthrxp_in
  .rx8b10ben_in (~0), // input wire [7 : 0] rx8b10ben_in
  .rxchbonden_in (gt_rx_chbonden_in), // input wire [7 : 0] rxchbonden_in
  .rxchbondi_in (gt_rx_chbondi_in), // input wire [39 : 0] rxchbondi_in
  .rxchbondlevel_in (gt_rx_chbondlevel_in), // input wire [23 : 0] rxchbondlevel_in
  .rxchbondmaster_in (gt_rx_chbondmaster_in), // input wire [7 : 0] rxchbondmaster_in
  .rxchbondslave_in (gt_rx_chbondslave_in), // input wire [7 : 0] rxchbondslave_in
  .rxcommadeten_in (~0), // input wire [7 : 0] rxcommadeten_in
  .rxmcommaalignen_in (~0), // input wire [7 : 0] rxmcommaalignen_in 
  .rxpcommaalignen_in (~0), // input wire [7 : 0] rxpcommaalignen_in
  .rxpolarity_in (gt_rx_polarity), // input wire [7 : 0] rxpolarity_in
  .tx8b10ben_in (~0), // input wire [7 : 0] tx8b10ben_in
  .txctrl0_in (0), // input wire [127 : 0] txctrl0_in
  .txctrl1_in (0), // input wire [127 : 0] txctrl1_in 
  .txctrl2_in (0), // input wire [63 : 0] txctrl2_in
  .gthtxn_out (), // output wire [7 : 0] gthtxn_out
  .gthtxp_out (), // output wire [7 : 0] gthtxp_out
  .gtpowergood_out (), // output wire [7 : 0] gtpowergood_out
  .rxbyteisaligned_out (gt_rx_aligned), // output wire [7 : 0] rxbyteisaligned_out
  .rxbyterealign_out (gt_rx_realign), // output wire [7 : 0] rxbyterealign_out
  .rxcdrlock_out (), // output wire [7 : 0] rxcdrlock_out
  .rxchanbondseq_out (gt_rx_chanbondseq_out), // output wire [7 : 0] rxchanbondseq_out
  .rxchanisaligned_out (gt_rx_chanisaligned_out), // output wire [7 : 0] rxchanisaligned_out
  .rxchanrealign_out (gt_rx_chanrealign_out), // output wire [7 : 0] rxchanrealign_out
  .rxchbondo_out (gt_rx_chbondo_out), // output wire [39 : 0] rxchbondo_out
  .rxcommadet_out (), // output wire [7 : 0] rxcommadet_out
  .rxctrl0_out (gt_rxctrl0), // output wire [127 : 0] rxctrl0_out
  .rxctrl1_out (gt_rxctrl1), // output wire [127 : 0] rxctrl1_out
  .rxctrl2_out (gt_rxctrl2), // output wire [63 : 0] rxctrl2_out
  .rxctrl3_out (gt_rxctrl3), // output wire [63 : 0] rxctrl3_out
  .rxpmaresetdone_out (), // output wire [7 : 0] rxpmaresetdone_out
  .txpmaresetdone_out () // output wire [7 : 0] txpmaresetdone_out
);

0 Kudos
1 Solution

Accepted Solutions
roym
Moderator
Moderator
2,400 Views
Registered: ‎07-30-2007

Yes the max skew is in bytes.  That explains it.




----------------------------------------------------------------------------
Don't forget to reply, kudo, and accept as solution
Be sure to visit the Resources post periodically to keep up with the latest
https://forums.xilinx.com/t5/Serial-Transceivers/Serial-Transceiver-Forum-Guidelines-and-Useful-Resources/td-p/1173590
----------------------------------------------------------------------------


View solution in original post

0 Kudos
11 Replies
roym
Moderator
Moderator
2,098 Views
Registered: ‎07-30-2007

What are byteisaligned and byterealign doing?  byterealign should only pulse one time and the byteisaligned should go high and stay high.  Sometimes there can be problems with the comma alignment which would keep the channel bonding from completing successfully.  It is usually faster to debug these problems in simulation.  Does the channel pass simulation?




----------------------------------------------------------------------------
Don't forget to reply, kudo, and accept as solution
Be sure to visit the Resources post periodically to keep up with the latest
https://forums.xilinx.com/t5/Serial-Transceivers/Serial-Transceiver-Forum-Guidelines-and-Useful-Resources/td-p/1173590
----------------------------------------------------------------------------


bkuschak
Adventurer
Adventurer
2,092 Views
Registered: ‎10-01-2013

Yes, these are operating as expected. I captured data using the ILA.

byteisaligned is constant 0xFF.  

byterealign is constant 0 for the time segment I'm looking at.

 

One thing that I noticed is that my source device is sending disparity errors on some lanes when it sends the CB sequence.  The lanes that have disparity errors do not bond correctly.  That led me to try setting RX_DISPERR_SEQ_MATCH = FALSE.  

 

This thread was marked as spam after I posted it, so yesterday I posted a separate thread asking about RX_DISPERR_SEQ_MATCH. 

0 Kudos
bkuschak
Adventurer
Adventurer
2,049 Views
Registered: ‎10-01-2013

OK, I think I found the problem. My skew is larger than CHAN_BOND_MAX_SKEW can accommodate. (Most of this seems to be due to the source device, not the channel.)  As noted above, I'm seeing six RXUSERCLK2 worse case skew.


I was interpreting the UG576 explanation of CHAN_BOND_MAX_SKEW as designated in units of RXUSRCLK, as to provide a maximum of 14 RXUSRCLK of deskew capability.  But I see now that this was incorrect.  Based on simulation it appears this really is limited to 14 bytes, not 14 clocks.

 

Can Xilinx clarify the UG576 statement below for these settings:

  • 32 bit interface width
  • RX_DATA_WIDTH = 40
  • RX_INT_DATAWIDTH = 1
  • RXUSRCLK2 = RXUSRCLK, and output 32 bits per RXUSRCLK2

 

CHAN_BOND_MAX_SKEW:
This attribute controls the number of USRCLK cycles
that the master waits before ordering the slaves to
execute channel bonding. This attribute determines
the maximum skew that can be handled by channel
bonding. It must always be less than one-half the
minimum distance (in bytes or 10-bit codes) between
channel bonding sequences. Valid values range from 1
to 14.

 

Please confirm: the transceivers can accept a maximum of three 32-bit words of skew (not 14), as seen at the GTH output gtwiz_userdata_rx_out.

 

channel_bonding_sim2_works_with_3_word_skew - Copy.PNG

 

channel_bonding_sim3_fails_with_4_word_skew - Copy.PNG

 

 

 

roym
Moderator
Moderator
2,401 Views
Registered: ‎07-30-2007

Yes the max skew is in bytes.  That explains it.




----------------------------------------------------------------------------
Don't forget to reply, kudo, and accept as solution
Be sure to visit the Resources post periodically to keep up with the latest
https://forums.xilinx.com/t5/Serial-Transceivers/Serial-Transceiver-Forum-Guidelines-and-Useful-Resources/td-p/1173590
----------------------------------------------------------------------------


View solution in original post

0 Kudos
die@matrix
Visitor
Visitor
255 Views
Registered: ‎05-28-2019

I'm trying to do the same as you (as it seems) - down to the values of the channel bonding - encountering the same problems.

May I ask how you managed the situation in the end?
Is there an alternative (like overriding the generics to higher values) to writing your own deskew logic and spending fabric resources on it?

It seems strange that the skew is so high, routing won't be the problem since 14 bytes make up a lot of centimetres
=> Seems like the sender already has a high skew (for whatever reason) you have to compensate in the end.

0 Kudos
bkuschak
Adventurer
Adventurer
206 Views
Registered: ‎10-01-2013

My implementation changed significantly since that post 3 years ago. Another complication not listed above is that the source device was found to send an odd number of bytes between commas.  I wrote my own deskew logic in the fabric, and switched from 32 bit to 16 bit data path in the GTH.  Works fine now.

0 Kudos
die@matrix
Visitor
Visitor
199 Views
Registered: ‎05-28-2019

Thanks for the advice, we just observed the same: an END_CODE (containing a comma at the start) was placed unaligned at the end of the line and got repeated after some bytes (by the transceiver) since we told it to align commas to 4 byte boundaries.
=> We will surely see some more funny details before we (hopefully) get it to work but it's nice to know that others did succeed in the end.
=> Switching to 16b will be a bit too fast for our Artix7 -> is there a "no go" for the 32b-path? Surely everything is do-able but is there some strong resistance, we should look out for?

0 Kudos
bkuschak
Adventurer
Adventurer
182 Views
Registered: ‎10-01-2013

Using a 32 bit data path just means that your protocol FSM will need to handle incoming symbols in any one of 4 alignment positions.  With 16 bit processing you only need to handle 2 alignment positions.  Surely it's possible to do it either way.  For me, the clock rate <120 MHz was easy to handle in the Ultrascale fabric.  Maybe more difficult in Artix7?

0 Kudos
die@matrix
Visitor
Visitor
172 Views
Registered: ‎05-28-2019

I thought, we could make the world easy for us using Xilinx' Align_To-Feature...
It "seems" to work in a way:
* the line ending that we would like to look like FDFEFDBC will come in with a leading 0x5b (in the screenshot below)
* then some transceiver magic happens and we get the re-aligned word in the next clock... 

 behaviour for misaligned line endingsbehaviour for misaligned line endings

=> it will effectively create 1-3 bytes by a copy if need be and resume its work aligned again.

I cannot foresee at the moment if that is enough for a more or less stable version. The additional clock that happens in here might leave the lanes in mis-alignment but fortunately the code directly after a line ending is the DESKEW code

0 Kudos
bkuschak
Adventurer
Adventurer
154 Views
Registered: ‎10-01-2013

In this case you may need to add some logic to properly detect the end of line condition after the fact, and terminate your data on the correct cycle.

If I recognize the protocol you are decoding, recall that subsequent IDLE codes may be any number of bytes in length.  If you are correctly 32-bit aligned after END and DESKEW, you may still become misaligned at the next START code.  This timing may vary depending on the specific options that you have configured in your source device.

0 Kudos
die@matrix
Visitor
Visitor
139 Views
Registered: ‎05-28-2019

That is some good advice.
We'll keep the IDLE codes in mind and have an extra eye on them.
For now, the deskew in logic seems to make some progress and we'll tackle the next bumbs one at a time

Thank you for sharing your experiences, it's always nice to know which ways others
in the same / similar position went on (and if that road still pleases them).

Best regards,
     Roman

0 Kudos