UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Visitor iansherwood
Visitor
471 Views
Registered: ‎07-06-2018

Multiple Block design JESD204B Ultrascale design

I am working on an Ultrascale xcku060 design with 4 ADC’s connected with 4 lanes of 4.8Gbps JESD204B each to banks 224, 225, 227 and 228. The design is implemented using 4 block designs with each PHY mapped to its own bank. 

There are refclks connected to banks 224 and 228 (the 224 refclk for the 224 and 225 GTH’s and the 228 for the 227 and 228 GTH’s).

I can get any 2 ADC's out of the 4 to do full link synchronization and get valid data across the link but when I enable any 3rd (or 4th) device the link fails and I cannot even get comma detection (powergood, reset_done and cpll_lock all still seem okay). I have enabled an IBERT module and with 2 blocks in the design I can see a normal eye but with a third attached the eye completely collapses. I have checked all the FPGA and GTH and ADC power supplies for noise and these seem okay. The clocks and comma signals on the links look stable. I can see that the Rx bufstatus signals indicate that the elastic buffers are under running and over flowing when the links fail.

Is there any other debug signals that can help locate the source of the problem?   

 

0 Kudos
17 Replies
Visitor iansherwood
Visitor
461 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

I forgot to say that the design is using JESD204 PHY (4.0) and JESD204(7.2) modules with 4 lanes per link in receive only.

0 Kudos
Xilinx Employee
Xilinx Employee
431 Views
Registered: ‎08-10-2007

Re: Multiple Block design JESD204B Ultrascale design

Hello,

As you are seeing buffer overflow errors when the issue arises, can you increase the LMFC buffer in the receiver?  Does this make a difference?

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Visitor iansherwood
Visitor
425 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

Thanks for the reply - I will give it a go but I thought the comma detection would be before the LMFC buffer in the receiver signal chain?
Thanks
Ian
0 Kudos
Xilinx Employee
Xilinx Employee
410 Views
Registered: ‎10-19-2011

Re: Multiple Block design JESD204B Ultrascale design

Hi @iansherwood,

did you check the GT supplies with the full load? Is there any additional activity?
How much noise do you have there?

------------------------------------------------------------------------------
Don't forget to reply, give kudo and accept as solution
------------------------------------------------------------------------------
0 Kudos
Visitor iansherwood
Visitor
364 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

20190122_164437.jpg

 

 

I have probed right on the decoupling caps mounted within the FPGA outline (on the underside of the board) and I can see no discernable difference in the noise between 2 ADC's working and 3 not working. This photo is typical of the sort of noise I am seeing on the MGTAVTT, MGTAVCC and MGTVCCAUX rails.

The design is not using any MGT transmitters only receivers. The gt_tx inputs to the PHY are floating - I did try tying them lo in case this was a problem but this did not seem to have any effect.

The LMFC buffer is set to the maximum size of 1024 and the has no effect on the failure.

0 Kudos
Xilinx Employee
Xilinx Employee
358 Views
Registered: ‎08-10-2007

Re: Multiple Block design JESD204B Ultrascale design

There are 2 registers in particular you could take a look at to get more information about the problem.  These are Link Error Status (0x01C) and Debug Status (0x03C).  You can find info on the Link Error Status register on page 32 of PG066 and info on the Debug Status register on page 35 of PG066.

Also, as you are using the PHY, I presume the rxencommalign signal is connected between the PHY and RX, and between all PHYs?

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Visitor iansherwood
Visitor
343 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

I will take a look at those registers.

I do have the rxencommalign connected between the JESD204 module and the JESD204PHY modules and I can see that this is being asserted.

The design has 4 JESD204 RX modules and 4 JESD204PHYs so there are 4  seperate rxencommalign signals each from a JESD204 module to a JESD204PHY. When you say that rxencommalign should be connected to all PHYs are you thinking of a design with one JESD204 module serving 4 JESD204PHYs? I do not think this is possible as a single JHESD204 module can only support up to 8 lanes - this design requires 16 lanes. 2 RX modules with 4 PHYs is possible but it is currently implemented as 4 RX and 4 PHY - is this an issue?

 

0 Kudos
Xilinx Employee
Xilinx Employee
333 Views
Registered: ‎08-10-2007

Re: Multiple Block design JESD204B Ultrascale design

If you are using multiple PHYs, the rxencommaalign signal should be connected between the 4 PHYs.  Do you have that setup in your design?

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Visitor iansherwood
Visitor
329 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

block.jpg

0 Kudos
Visitor iansherwood
Visitor
328 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

The intention is to have 4 sets of the attached block diagram.

0 Kudos
Xilinx Employee
Xilinx Employee
251 Views
Registered: ‎08-10-2007

Re: Multiple Block design JESD204B Ultrascale design

Are these blocks completely independent, or do they interact with each other?

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Visitor iansherwood
Visitor
245 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

External to the FPGA each block is connected to a different ADC so they can operate independently.

0 Kudos
Visitor iansherwood
Visitor
134 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

Hi,

With 3 modules in the design :

The link error status reg (x01c)  and debug status register (x03c) and they both read back x00000000.

The readback does look functional as I read x07020400 from the version reg (0x000).

The power good and cpllock signals are both at '1'.

The rxcommadet(3:0) are showing x"0", x"4", x"8", x"0" ,x"c", x"e", x"a"

The rxbufstatus(2:0) as previuosly stated show "110", "000", "101" pattern repeat. 

 

With 2 modules in the design:

The link error status reg (x01c)  and debug status register (x03c) and they both read back x00000000.

The power good and cpllock signals are both at '1'.

The rxcommadet(3:0) are showing constant x"f"

The rxbufstatus(2:0) as previuosly stated show constant "000". 

 

0 Kudos
Xilinx Employee
Xilinx Employee
127 Views
Registered: ‎08-10-2007

Re: Multiple Block design JESD204B Ultrascale design

Hi, the data you have included for 2 modules vs 3 modules raises some questions.

If the system with 2 modules was working, I would expect to see the debug status register read something other than 0x0000000.  If valid data was detected, at least, you would see Start of Data was detected.  These are cleared on read, or when the core is reset, but are instantly set again.  

Also, it seems strange that you are not seeing errors on the link error status register when you have 3 modules in the design.

Can you check the data being outputted when the 2 modules are up and running, is it what you would expect?

Can you set the 2 modules running (and outputting data as you expect), then turn on the 3rd module, and check what changes?  i.e. have 2 modules up and running for some time, before adding a 3rd, rather than turn 3 on together?

What clocking scheme have you implemented in your design?  Are you following Figure 3-1 or Figure 3-3 of PG066?  

For the line rate you are using, what do you have the refclk set to?

Finally, for both the 2 module and 3 module scenario, can you capture the SYNC status register value?

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Xilinx Employee
Xilinx Employee
115 Views
Registered: ‎10-19-2011

Re: Multiple Block design JESD204B Ultrascale design

Hi @iansherwood,

to chime in additionally, the test to switch on a third data path later could also show some changes on supplies as you could trigger on the switch to see what the supplies are doing at this time.

To get back to your scope shot. It looks like you triggered on the noise on the line. Could you do a persistent measurement over a longer time without trigger?
On the picture shown you are already over 20mVpkpk noise, but it seem to be higher frequencies.
Could you measure in the range of 10kHZ-80MHz? That would be the range where you have to be below 10mVpkpk.

Could it also be that something on the ADC side changes? I am wondering about the bufferstatus going into over or underflow. JESD is a synchronous setup. There should be no drift in the buffer. You would have to have different frequencies on both sides of the buffer to get to this.
Is it possible that the ADCs do start the data stream differently when 3 ADCs are started up simultaneously?
If the ADC do not output data and the transceiver RX gets out of reset the CDR will not have anything to lock to. In this situation you could run into buffer over and underflow.

------------------------------------------------------------------------------
Don't forget to reply, give kudo and accept as solution
------------------------------------------------------------------------------
0 Kudos
Visitor iansherwood
Visitor
54 Views
Registered: ‎07-06-2018

Re: Multiple Block design JESD204B Ultrascale design

Hi @eschidl ,

Please find attached some scope screenshots with infinite persistance at lower frequencies. These have been measured as close to the FPGA balls as I can get. As you previously mentioned there is some high frequency noise - if I add a 20MHz filter on the scope the noise level drops well below the 10mV. I have tried adding some extra 100n decoupling under the FPGA on these rails but so far this had no effect.

1V2 rail (4us/div)

1v2.jpg

 

1V2 rail (200ns/div)

1v2b.jpg


1V rail (4us/div)

1v8pwr.jpg

1V8 rail (200ns/div)

1v8pwrb.jpg

 

1V rail (4us/div)

1vgthf_b.jpg

0 Kudos
Xilinx Employee
Xilinx Employee
18 Views
Registered: ‎10-19-2011

Re: Multiple Block design JESD204B Ultrascale design

Hi @iansherwood,

your supplies look relatively good. So I don't think the noise there will be an issue.
I assume that it looks the same, when you have 3 links running.

Did you by chance do a test starting up with just 2 links and then ramp up a third one later on?

What is the behaviour on the supplies then?

------------------------------------------------------------------------------
Don't forget to reply, give kudo and accept as solution
------------------------------------------------------------------------------
0 Kudos