UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Visitor iansherwood
Visitor
1,074 Views
Registered: ‎07-06-2018

Multiple Block design JESD204B Ultrascale design

I am working on an Ultrascale xcku060 design with 4 ADC’s connected with 4 lanes of 4.8Gbps JESD204B each to banks 224, 225, 227 and 228. The design is implemented using 4 block designs with each PHY mapped to its own bank. 

There are refclks connected to banks 224 and 228 (the 224 refclk for the 224 and 225 GTH’s and the 228 for the 227 and 228 GTH’s).

I can get any 2 ADC's out of the 4 to do full link synchronization and get valid data across the link but when I enable any 3rd (or 4th) device the link fails and I cannot even get comma detection (powergood, reset_done and cpll_lock all still seem okay). I have enabled an IBERT module and with 2 blocks in the design I can see a normal eye but with a third attached the eye completely collapses. I have checked all the FPGA and GTH and ADC power supplies for noise and these seem okay. The clocks and comma signals on the links look stable. I can see that the Rx bufstatus signals indicate that the elastic buffers are under running and over flowing when the links fail.

Is there any other debug signals that can help locate the source of the problem?   

 

0 Kudos
21 Replies
Visitor iansherwood
Visitor
1,064 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

I forgot to say that the design is using JESD204 PHY (4.0) and JESD204(7.2) modules with 4 lanes per link in receive only.

0 Kudos
Moderator
Moderator
1,034 Views
Registered: ‎08-10-2007

Re: Multiple Block design JESD204B Ultrascale design

Hello,

As you are seeing buffer overflow errors when the issue arises, can you increase the LMFC buffer in the receiver?  Does this make a difference?

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Visitor iansherwood
Visitor
1,028 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

Thanks for the reply - I will give it a go but I thought the comma detection would be before the LMFC buffer in the receiver signal chain?
Thanks
Ian
0 Kudos
Xilinx Employee
Xilinx Employee
1,013 Views
Registered: ‎10-19-2011

Re: Multiple Block design JESD204B Ultrascale design

Hi @iansherwood,

did you check the GT supplies with the full load? Is there any additional activity?
How much noise do you have there?

------------------------------------------------------------------------------
Don't forget to reply, give kudo and accept as solution
------------------------------------------------------------------------------
0 Kudos
Visitor iansherwood
Visitor
967 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

20190122_164437.jpg

 

 

I have probed right on the decoupling caps mounted within the FPGA outline (on the underside of the board) and I can see no discernable difference in the noise between 2 ADC's working and 3 not working. This photo is typical of the sort of noise I am seeing on the MGTAVTT, MGTAVCC and MGTVCCAUX rails.

The design is not using any MGT transmitters only receivers. The gt_tx inputs to the PHY are floating - I did try tying them lo in case this was a problem but this did not seem to have any effect.

The LMFC buffer is set to the maximum size of 1024 and the has no effect on the failure.

0 Kudos
Moderator
Moderator
961 Views
Registered: ‎08-10-2007

Re: Multiple Block design JESD204B Ultrascale design

There are 2 registers in particular you could take a look at to get more information about the problem.  These are Link Error Status (0x01C) and Debug Status (0x03C).  You can find info on the Link Error Status register on page 32 of PG066 and info on the Debug Status register on page 35 of PG066.

Also, as you are using the PHY, I presume the rxencommalign signal is connected between the PHY and RX, and between all PHYs?

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Visitor iansherwood
Visitor
946 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

I will take a look at those registers.

I do have the rxencommalign connected between the JESD204 module and the JESD204PHY modules and I can see that this is being asserted.

The design has 4 JESD204 RX modules and 4 JESD204PHYs so there are 4  seperate rxencommalign signals each from a JESD204 module to a JESD204PHY. When you say that rxencommalign should be connected to all PHYs are you thinking of a design with one JESD204 module serving 4 JESD204PHYs? I do not think this is possible as a single JHESD204 module can only support up to 8 lanes - this design requires 16 lanes. 2 RX modules with 4 PHYs is possible but it is currently implemented as 4 RX and 4 PHY - is this an issue?

 

0 Kudos
Moderator
Moderator
936 Views
Registered: ‎08-10-2007

Re: Multiple Block design JESD204B Ultrascale design

If you are using multiple PHYs, the rxencommaalign signal should be connected between the 4 PHYs.  Do you have that setup in your design?

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Visitor iansherwood
Visitor
932 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

block.jpg

0 Kudos
Visitor iansherwood
Visitor
930 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

The intention is to have 4 sets of the attached block diagram.

0 Kudos
Moderator
Moderator
853 Views
Registered: ‎08-10-2007

Re: Multiple Block design JESD204B Ultrascale design

Are these blocks completely independent, or do they interact with each other?

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Visitor iansherwood
Visitor
847 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

External to the FPGA each block is connected to a different ADC so they can operate independently.

0 Kudos
Visitor iansherwood
Visitor
736 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

Hi,

With 3 modules in the design :

The link error status reg (x01c)  and debug status register (x03c) and they both read back x00000000.

The readback does look functional as I read x07020400 from the version reg (0x000).

The power good and cpllock signals are both at '1'.

The rxcommadet(3:0) are showing x"0", x"4", x"8", x"0" ,x"c", x"e", x"a"

The rxbufstatus(2:0) as previuosly stated show "110", "000", "101" pattern repeat. 

 

With 2 modules in the design:

The link error status reg (x01c)  and debug status register (x03c) and they both read back x00000000.

The power good and cpllock signals are both at '1'.

The rxcommadet(3:0) are showing constant x"f"

The rxbufstatus(2:0) as previuosly stated show constant "000". 

 

0 Kudos
Moderator
Moderator
729 Views
Registered: ‎08-10-2007

Re: Multiple Block design JESD204B Ultrascale design

Hi, the data you have included for 2 modules vs 3 modules raises some questions.

If the system with 2 modules was working, I would expect to see the debug status register read something other than 0x0000000.  If valid data was detected, at least, you would see Start of Data was detected.  These are cleared on read, or when the core is reset, but are instantly set again.  

Also, it seems strange that you are not seeing errors on the link error status register when you have 3 modules in the design.

Can you check the data being outputted when the 2 modules are up and running, is it what you would expect?

Can you set the 2 modules running (and outputting data as you expect), then turn on the 3rd module, and check what changes?  i.e. have 2 modules up and running for some time, before adding a 3rd, rather than turn 3 on together?

What clocking scheme have you implemented in your design?  Are you following Figure 3-1 or Figure 3-3 of PG066?  

For the line rate you are using, what do you have the refclk set to?

Finally, for both the 2 module and 3 module scenario, can you capture the SYNC status register value?

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Xilinx Employee
Xilinx Employee
717 Views
Registered: ‎10-19-2011

Re: Multiple Block design JESD204B Ultrascale design

Hi @iansherwood,

to chime in additionally, the test to switch on a third data path later could also show some changes on supplies as you could trigger on the switch to see what the supplies are doing at this time.

To get back to your scope shot. It looks like you triggered on the noise on the line. Could you do a persistent measurement over a longer time without trigger?
On the picture shown you are already over 20mVpkpk noise, but it seem to be higher frequencies.
Could you measure in the range of 10kHZ-80MHz? That would be the range where you have to be below 10mVpkpk.

Could it also be that something on the ADC side changes? I am wondering about the bufferstatus going into over or underflow. JESD is a synchronous setup. There should be no drift in the buffer. You would have to have different frequencies on both sides of the buffer to get to this.
Is it possible that the ADCs do start the data stream differently when 3 ADCs are started up simultaneously?
If the ADC do not output data and the transceiver RX gets out of reset the CDR will not have anything to lock to. In this situation you could run into buffer over and underflow.

------------------------------------------------------------------------------
Don't forget to reply, give kudo and accept as solution
------------------------------------------------------------------------------
0 Kudos
Visitor iansherwood
Visitor
656 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

Hi @eschidl ,

Please find attached some scope screenshots with infinite persistance at lower frequencies. These have been measured as close to the FPGA balls as I can get. As you previously mentioned there is some high frequency noise - if I add a 20MHz filter on the scope the noise level drops well below the 10mV. I have tried adding some extra 100n decoupling under the FPGA on these rails but so far this had no effect.

1V2 rail (4us/div)

1v2.jpg

 

1V2 rail (200ns/div)

1v2b.jpg


1V rail (4us/div)

1v8pwr.jpg

1V8 rail (200ns/div)

1v8pwrb.jpg

 

1V rail (4us/div)

1vgthf_b.jpg

0 Kudos
Xilinx Employee
Xilinx Employee
620 Views
Registered: ‎10-19-2011

Re: Multiple Block design JESD204B Ultrascale design

Hi @iansherwood,

your supplies look relatively good. So I don't think the noise there will be an issue.
I assume that it looks the same, when you have 3 links running.

Did you by chance do a test starting up with just 2 links and then ramp up a third one later on?

What is the behaviour on the supplies then?

------------------------------------------------------------------------------
Don't forget to reply, give kudo and accept as solution
------------------------------------------------------------------------------
0 Kudos
Visitor iansherwood
Visitor
597 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

Hi,

I have tried instanciating three module and taking one of them in and out of reset (tx and rx_sys_reset to the phy and rx_reset to the jesd module).   The 2 that are not being reset still do not comma detect even with the 3rd in reset.  Is this what you mean by ramp up the 3rd later on?

While doing this I cannot see any change on the power rails. 

Thanks

 

0 Kudos
Xilinx Employee
Xilinx Employee
587 Views
Registered: ‎10-19-2011

Re: Multiple Block design JESD204B Ultrascale design

Hi @iansherwood ,

yes, that was the sequence I would have in mind. Only thing is with the tx_sys_reset and rx_sys_reset you will not power down the transceiver. It will still be running and only get a reset when you release the signal. For a power down you will need to control the power down signals of the transceiver directly. TXPD and RXPD you can get access to when you select the transceiver control and status ports.
For CPLLPD it is a bit more difficult. It is not brought up this way. You have to find cpllpd_in inside the PHY core and control this signal to power down the CPLLs. The reset signals of the core will only pulse this signal for a short time.

------------------------------------------------------------------------------
Don't forget to reply, give kudo and accept as solution
------------------------------------------------------------------------------
0 Kudos
Visitor iansherwood
Visitor
538 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

three_cpll_pd.png

 

Hi - I have tried all of your suggestions but none seem to have any effect on the problem. The above picture shows the result of holding the cpllpd pins high for the transciever jesd_1, while jesd_2 and jesd_3 are running. As you can see cpll_1 has shutdown as expected but jesd_2 and jesd_3 are still failing to detect commas.

0 Kudos
Moderator
Moderator
504 Views
Registered: ‎08-10-2007

Re: Multiple Block design JESD204B Ultrascale design

There is something strange going on here.  As mentioned, if the system with 2 modules was working, I would expect to see the debug status register read something other than 0x0000000.  If valid data was detected, at least, you would see Start of Data was detected.  These are cleared on read, or when the core is reset, but are instantly set again.  

Also, it seems strange that you are not seeing errors on the link error status register when you have 3 modules in the design.

Can you check the data being outputted when the 2 modules are up and running, is it what you would expect?

Can you set the 2 modules running (and outputting data as you expect), then turn on the 3rd module, and check what changes?  i.e. have 2 modules up and running for some time, before adding a 3rd, rather than turn 3 on together?

Can you also please clarify what clocking scheme you have implemented in your design?  Are you following Figure 3-1 or Figure 3-3 of PG066?  

For the line rate you are using, what do you have the refclk set to?

Finally, for both the 2 module and 3 module scenario, can you capture the SYNC status register value?

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos