cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Participant
Participant
956 Views
Registered: ‎05-10-2017

SERDES clock tolerance

How do I determine, how much clock skew can I have between the two 100MHz clocks going to FPGA_A and FPGA_B in the picture shown? 

I need deterministic latency on this link.  i.e. the number of clock periods to transfer data from FPGA_A to FPGA_B should be constant.  I am particularly worried that just as SERDES delays will be affected by PVT, so will the clock delays both inside and outside the chip.  

I have two Artix 7 FPGAs with a unidirectional serdes link as shown in the attached picture.  FPGA_A has SERDES TX and FPGA_B has SERDES RX.  These use LVDS I/Os.  I use IDELAYCTRL blocks.  So I think that the SERDES lanes will be compensated for PVT variations.  It is also important to note, that serdes data and clock are provided to FPGA_B over a 10ft cable. 

 Annotation 2019-10-10 153328.png

 

Tags (4)
0 Kudos
13 Replies
Highlighted
Adventurer
Adventurer
889 Views
Registered: ‎05-07-2018

Re: SERDES clock tolerance

Hi

the serial interface is not capable of deterministic latency. I read some papers about this, and the authors mentioned PPM must be less than 100.

But the most important thing is : There is bit slip in Serdes and that causes inconsistency on latency. 

If you can align phase and compensate bit slip, Maybe you will reach your purpose. 

0 Kudos
Highlighted
Teacher
Teacher
872 Views
Registered: ‎07-09-2009

Re: SERDES clock tolerance

Have  look at a standard SerDes blocks , such as Aurora, 

    https://www.xilinx.com/products/intellectual-property/aurora64b66b.html

 

All of them work asyncronous, 

     The clock at the recever has no reference to the phase of the transmitter. 

         they can also allow for the inherrant small difference betwene the Tx and Rx clock,

 

The protocol adds extra information to the sent data at the transmitter, that the reciever can recognise that it can remove. Extra data is also used for things like marking start / stop of frames . The protocole is also used for thigns like ensuring sufficient information for the receive clock to be recovered and to ensure there is no DC bias in the data stream.  

The tx and Rx fifo ensure the data rate is constrained averagly at the same rate, no data lost.

I have recolection of syncronous protocoles , I think the orriginal ATM link was. But thats 30 years ago.

 

I dont know if you can send data without using a protocole, I cant see how you would ensure the above,

 

If you want to ensure latency between multiple links is predicatble and constant, 

    then try using the JESD 204 link  protocole,

       its used in ADC / DAC links, where the data HAS to arrive at the far end at a constant rate ,

 

<== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
0 Kudos
Highlighted
Participant
Participant
849 Views
Registered: ‎05-10-2017

Re: SERDES clock tolerance

Hmm...

Thank you for your answer.

I am not seeing why in a synchronous system bitslip will cause uncertainty of 1 clock period.  Can you please point to some references that can help me understand?

Thank you.

0 Kudos
Highlighted
Adventurer
Adventurer
825 Views
Registered: ‎05-07-2018

Re: SERDES clock tolerance

Hi

I had some issues with this before. but I am sure about bit slip function, I need to reach deterministic latency.

https://forums.xilinx.com/t5/Serial-Transceivers/Phase-alignment-with-rxslide-in-PMA-mode-KC705/m-p/1022496#M5519

I think RX Clock and Tx clock have (0 to 360)-degree difference on the base of bit slip amount, RX clock will have (0 to 360) degree difference with TX Clock. and it causes 1 clock uncertainty in latency. 

hope this be helpful for you

0 Kudos
Highlighted
Teacher
Teacher
808 Views
Registered: ‎07-09-2009

Re: SERDES clock tolerance

JESD 204 solves deterministic latency

Is there any reason you can not use proven technology ?

You mention that the system your proposing is synchronous,
Id say in reality its not.

Synchronous implies that the Tx and the Rx are running to the identical clock. This includes if the Tx end goes up say 10 PPM, so does the receive end.

Ah you say, both ends have the same clock,
but they dont.

Remember inside the Tx is a bunch of PLL, that multiple the MHz clock up to many GHz, to send out the data on the ser des line,

In the receiver, there is a clock recovery circuit, that uses the MHz **bleep** as its reference, then multiply it up using its bunch of PLL, which is then phase shifted to extract the received data in the centre of the receive eye.

Thats a bunch of phase locked loops , and phase snifters away from a synchronous system,

Suggest you draw it out, add in say 100 PPM jitter to each PLL, see what you end up at with say a 12 Gb/s link...

Thats why there is bit slip, extra data added to the stream and the fifos, to take care of what is really an asynchronous system across the link,

JESD 204 takes care of this,
it assumes that the bit in the middle is asynchronous, Effectively , JESD 204 B sends frame markers over the ser des link and a time marker at the same frame rate ( which is only at the MHz rate ) between the Tx and the Rx,

Then a controller system has a bit of a chat ( over SPI ) . The up come is that the link has guaranteed and repeatable latency

<== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
0 Kudos
Highlighted
Guide
Guide
778 Views
Registered: ‎01-23-2009

Re: SERDES clock tolerance

First, lets make sure we are all on the same page. The original post clearly mentioned PLL, LVDS, IDELAYCTRL, so I am pretty sure these are not using the gigabit transcievers (GTX/GTH), but are using the ISERDES/OSERDES. Some of the responses talking about Aurora and JESD204 are for gigabit transceivers...

The issue here is not about clock tolerance (really clock skew) it is about the entire interface.

Regardless of whether you are using the ISERDES/OSERDES, the IDDR/ODDR or conventional flip-flops (in the IOB or elsewhere) what you are describing here is a system synchronous interface - both devices get a common clock and try and communicate synchronously with that common clock. This system must be analyzed as a complete system.

You don't say if this is SDR or DDR, so I am going to assume SDR (thus the interface is running at 100Mbps/pin).

In essence, you have a timing path that launches at the rising edge of the 100MHz clock in the transmitting FPGA and is captured at the next rising edge of the 100MHz clock in the receiving FPGA (I am assuming that the PLLs in both FPGAs are running at 1:1, so the interface is really 100Mbps/pin, not some multiple).

The "common point" for this system is the CLK_GEN. From here, we need to add up all the delays

  • Source Clock Delay (SCD)
    • From CLK_GEN to FPGA_A pin (I am assuming a clock capable pin)
    • through the IBUF in FPGA_A
    • through the PLL in FPGA_A
    • through a global clock buffer in FPGA_A
    • through the global clock tree in FPGA_A
  • Datapath Delay (DPD)
    • through the OSERDES in FPGA_A
    • though the OBUF in FPGA_A
    • through the 10ft cable betwen FPGA_A and FPGA_B (I will come back to this)
    • through the IBUF in FPGA_B
    • to the ISERDES in FPGA_B
  • Destination clock delay (DCD)
    • From the CLK_GEN
    • through the 10ft cable to FPGA_B
    • through the IBUF of FPGA_B
    • through the PLL of FPGA_B
    • through the BUFG of FPBA_B
    • through the global clock network of FPGA_B
    • to the ISERDES of FPGA_B
    • add the setup requirement of the ISERDES

For this entire path, the following equation must be true

SCD(max) + DPD(max) <= 10ns + DCD(min) - jitter(max)

Now, lets get to the 10ft cable - are you even sure the I/O in question is capable of driving a 10ft cable at 100MHz? It's not clear that the answer to this is yes. Even if you can, this is going to introduce a large delay on both of the paths through the cable; a cable usually propagates at around 1/2 the speed of light, which is about 0.5ns/ft - therefore your cable has a delay of about 20ns - this is more than twice your bit period.

Now, you have a 10ft cable on both the DPD and the DCD, so these will cancel somewhat, but even if these cancel to with 10%, this adds 2ns of uncertainty to this path. The uncertainties on the rest of the long list of items above will almost certainly add more than the remaining 8ns of uncertainty allowed in this system, which means that this system will almost certainly not meet timing. And this is at 100Mbps - at anything higher it is almost certainly out of the question.

On top of this, you have the question of framing; the transmitter is taking parallel data and converting it to serial, the receiver is doing the opposite. Without any explicit mechanism, there is no way for the receiver to know which bit in the serial stream is the most significant bit of the original parallel word. This has nothing to do with clock skew or latencies, but about how the FPGAs came out of reset; each of them somewhere is maintaining a counter of which bit it is sending/receiving next; these counters count from 0 to 9, but without some mechanism of synchronizing them, they will end up reconstructing the parallel word at the receiver incorrectly.

In short, this system is likely not viable. It is specifically for systems like this (sending data over long cables) that we have things like the Gigabit transcievers running protocols like Aurora (or other encoded formats) - to handle the long transmission times over a cable (using clock/data recovery). However, these have no latency guarantees nor predictable latencies; you need to have an even higher level of protocol (like JESD204) to control latency, and it does this with additional parallel signals and embedded sequences...

(And just a quick note, the IDELAYCTRL is only required to calibrate the IDELAY/ODELAY, it has no effect on anything else, including the ISERDES/OSERDES).

Avrum

Highlighted
Teacher
Teacher
762 Views
Registered: ‎07-09-2009

Re: SERDES clock tolerance

My bad, missed it was using the IO SerDes not the GTx SerDes,
<== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
0 Kudos
Highlighted
Participant
Participant
757 Views
Registered: ‎05-10-2017

Re: SERDES clock tolerance

Avrum,

Thank you for the response.  And thank you for narrowing it down to my application i.e. LVDS IO SERDES. 

We have 4:1 serialization factor.  SDR.  So the serial clock rate is 400MHz.  The data stream is encoded.  We use IDELAYCTRL + DELAY TAPS to bit align.  Subsequently, we do word alignment with bitslip.  I would not like to increase the cost of the solution by using JESD204 if I do not have to.  

We have cable drivers and receivers on the board that helps with the 10ft cable transmission.  The prototype system is working in the lab.  You are correct that most of the skew between the clock and data will cancel out in the cable as the same cable is used to carry those signals.  Further, my assumption is that the clocks after the PLL will be compensated so the contribution to skew is only from IBUFG to PLL.  

  1. If I determine a certain phase relationship between TX CLK and RX CLK, can I ascertain deterministic latency? 
  2. Can the bitslip logic + IDELAYCTRL cause an uncertainly of over a clock period?

Take a look at this old paper.  It talks about XCVRs but the premise I believe can be applicable to SERDES IO blocks as well.  It seems to indicate that deterministic latency can be achieved in a synchronous system. 

0 Kudos
Highlighted
Participant
Participant
756 Views
Registered: ‎05-10-2017

Re: SERDES clock tolerance

Thank you for your continued interest.  JESD204 links increase the cost in the system for me.  And as you pointed out, my application uses SERDES IO blocks and not XCVRs.

0 Kudos
Highlighted
Adventurer
Adventurer
686 Views
Registered: ‎05-07-2018

Re: SERDES clock tolerance

Hi

in this paper "High-Speed, Fixed-Latency Serial Links With FPGAs for Synchronous Transfers " 

authors make some changes on IP core and they reach deterministic latency. they worked on the BitSlip issue as I mentioned in my last comment. 

I think the solution is similar. you must compensate bit slip.

Good luck

0 Kudos
Highlighted
Adventurer
Adventurer
679 Views
Registered: ‎05-07-2018

Re: SERDES clock tolerance

my guess is
you must shift the Clock according to the amount of (bit_slip*360/data_width) to align the phase between transceiver and receiver.
0 Kudos
Highlighted
Participant
Participant
616 Views
Registered: ‎05-10-2017

Re: SERDES clock tolerance

I am still not clear on how much cell skew we can tolerate if the idelayctrl and bitslip is enabled.  Maybe someone who knows how the bitslip and idelay blocks work together can help.

The confusion is, if the clocks are synchronous and then ideally the system should provide deterministic latency.  But we are not clear how much margin we can have due to pvt.

0 Kudos
Highlighted
Community Manager
Community Manager
570 Views
Registered: ‎08-08-2007

Re: SERDES clock tolerance

Hi @shparekh 

 

If I understand the diagram correctly you are sending one clock to the FPGA A and FPGA B, when you transmit from FPGA A to FPGA B you are not sending a clock with the data. 

This would then be considered a Asynchronous data transfer by the FPGA as the clock is not coming with the data. 

The PVT tolerances will impact the data capture at the ISERDES. You need to meet a static timing anlaysis for your interface to have reliable capture of data. 

 

The latency numbers are documented in the SelectIO UG and are not impacted by PVT : https://www.xilinx.com/support/documentation/user_guides/ug471_7Series_SelectIO.pdf

 

OSERDES_Latency.PNGISERDES.PNG

Thanks,
Sandy

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos