cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
joe306
Scholar
Scholar
943 Views
Registered: ‎12-07-2018

Aurora 64B/66B gt_pll_lock toggling periodically

Hello, I'm using a Zynq Ultrascale+ MPSoC on a custom board and trying to use the Aurora 64B/66B IP. I have the Loopback set to Normal Mode. I have connected the gt_ref_clk to 156.25Mhz external oscillator and the init_clk to a Clock Wizard that generates 43Mhz. I am connecting the Aurora to a Fast Fiber Transceiver. I am modeling the Power Up reset as described on page 68 Figure 3-2.

The problem I am having is the gt_pll_lock keep toggling and I don't know why. Can anyone help me please?

 

Thank you,

Joe

0 Kudos
16 Replies
karnanl
Xilinx Employee
Xilinx Employee
896 Views
Registered: ‎03-30-2016

Hello Joe @joe306 

Let me add some comment on this post \.

1. I believe your Aurora 64B66B connectivity is something like this.
    Joe_Aurora.png

# If not could you please share a simple picture to show how you connected your system.
    Please share also your Aurora 64B66B XCI file ?

2. Could you please ensure that both 156.25MHz and 43MHz for init_clk are free-run and stable before you starting Aurora 64B66B

3. if PLL_LOCK is toggling, I would suspect 3 things
        (a) REFCLK or INIT_CLK does not follow clock setting set in the GUI
        (b) REFCLK source has a phase-noise performance
        (c) Aurora 64B66B IP is resetting itself because of a lot of errors.
             Do you have ILA waveform to share ? Could you please share the following signals during initialization ?
                - channel_up
                - lane_up
                - soft_err
                - hard_err
                - qpllreset_out
                - gt_powergood
                - pma_init
                - reset_pb
                - sys_reset_out

Thanks & regards
Leo


------------------------------------------------------------------------------------------------

Don’t forget to reply, kudo, and accept as solution.

If starting with Versal take a look at our Versal Design Process Hub and our
Versal Blogs

------------------------------------------------------------------------------------------------
joe306
Scholar
Scholar
857 Views
Registered: ‎12-07-2018

Hello, thank you very much for responding to my message.
Your diagram is exactly what I am doing.
I got this working on the ZCU106 board but not on my custom board.
REFCLK is from a Free Running oscillator and INIT_CLK is coming from a Clock Wizard.
I am following the Power On Sequence as described in Figure 3-2 page 68 of PG074.
I will get you the waveforms you mentioned on 3.c
I do I generate a XCI file? I will look around for the at file.
Thank you,
Joe
joe306
Scholar
Scholar
850 Views
Registered: ‎12-07-2018

Hello, I've routed some of the signals you requested to output pins:

Power UP ResetPower UP Reset

START2.jpg

START3.jpg

START4.jpg

 

START5.jpg

It looks like the link_reset_out is periodically being reset and that is causing or leading to the sys_reset_out to reset (go high) and causes the gt_pll_lock to go low.

I have some more to add in another post.

I've uploaded my design that you can regenerate the block design.

Respectfully,

Joe

0 Kudos
joe306
Scholar
Scholar
849 Views
Registered: ‎12-07-2018

Hello, I want to show you the init_clk flow:

init_clk_43mhzinit_clk_43mhz

init_clock_2.jpg

init_clock_3.jpg

init_clock_4.jpg

gt_refclk_inputgt_refclk_input

I am still a newbie here and I need to do better on the timing closure.

Joe

joe306
Scholar
Scholar
842 Views
Registered: ‎12-07-2018

Hello, are is the Aurora IP:

Aurora IPAurora IP

Floorplan of gt_refclkFloorplan of gt_refclk

Zoom in Quad LocationsZoom in Quad Locations

Pin I/O ViewPin I/O View

0 Kudos
joe306
Scholar
Scholar
841 Views
Registered: ‎12-07-2018

Hello, I was reading page 26 Table 2-9 about using MMCM for any clocks to the IP:

mmcm.jpg

Since I am using an MMCM for the init-clk do I need to use the mmcm_not_locked pin?

This pin is only available when in Include Shared Logic is selected. I'm not using that because I need the user_clk_out pin to drive the AXI DMA IP.

Thank you very helping me. I'll get back to work.

Respectfully,

Joe

0 Kudos
karnanl
Xilinx Employee
Xilinx Employee
815 Views
Registered: ‎03-30-2016

Hello @joe306 

Just adding my comments:

1. I believe your REFCLK is good. I can see that gt_pll_lock is asserted.

2. pma_init and reset_pb should be high on default setting (after board power-up),
    please deassert those signals only after you can ensure that gt_powergood is high.
    # According to your oscilloscope waveform, I suspecting that you don't follow this guideline.

3. Yes, your understanding is correct that link_reset_out assertion caused this  periodical gt_pll_lock deassertion.

4. My concern looking at oscilloscope waveform above is LANE_UP does not toggle at all.
    Aurora 64B66B IP has a watchdog timer to check LANE_UP status, if this signal is not asserted IP will reset itself periodically.

5. Joe:I got this working on the ZCU106 board but not on my custom board.

    This is a very good hint. perhaps the problem root-cause is not on your design but on your board.


So,
Could you please ensure that Fiber Optic Transceiver is actually sending a correct signal ( in term of DC spec, and line-rate ) to your GTH RX ?
Can you please capture the GTH RX input signal ?
If GTH RX is receiving signal correctly , I am expecting that LANE_UP should be asserted.


Regards
Leo


------------------------------------------------------------------------------------------------

Don’t forget to reply, kudo, and accept as solution.

If starting with Versal take a look at our Versal Design Process Hub and our
Versal Blogs

------------------------------------------------------------------------------------------------
joe306
Scholar
Scholar
796 Views
Registered: ‎12-07-2018

Hello, thank you very much for responding to my message. On the ZCU106 board init_clk is coming from an external free-running oscillator and I don't control the pma_init or the reset_pb. On my board the init_clk is coming from a Clock Wizard and I do control the pma_init and reset_pb. Last, on the ZCU106 I'm running the lanes near 10Gb on my custom board the fast fiber tops out at 4Gb so I am running the lanes at 3.125Gb.
Next, there is an I2C link to the Fast Fiber and when I read the I2C registers looking for a Loss of Signal bit, it always shows high indicator that there is a loss of signal on the loopback cable. So, that is a bummer, but perhaps I can run with LoopBack set to PMA mode. I will try another custom board next week. I know that when I put the custom board in near-end PMA or near-end PCS mode that I do get a Lane up. I can show you that next week.
Last, I'm going to follow your suggestion #2 and get that fixed next week.
Last, last do I need to connect to the mmcm_not_locked pin? I'm not doing that currently.

Thank you very much for helping me. I've been working on this problem for the last two weeks with not much progress.

Respectfully,
Joe
0 Kudos
karnanl
Xilinx Employee
Xilinx Employee
719 Views
Registered: ‎03-30-2016

Hello @joe306 

1. You definetely need to connect mmcm_not_locked signal as described in PG074.
mmcm_not_locked.png

2. "there is an I2C link to the Fast Fiber and when I read the I2C registers looking for a Loss of Signal bit, it always shows high indicator that there is a loss of signal on the loopback cable."

This is a good hint.
So, perhaps my assumption was correct that Aurora 64B66B signal from TX is not properly loopback-ed.
Please let me know if you still see issue with Near-End PMA loopback.

3. One more thing to check is the "Transceiver setting".
Did you configure "Insertion loss" and "Equalization mode" to fit your board configuration ?
If you are aware of channel insertion loss value between "Fabric Optic Transceiver" and "GTH RX" , you need to set correct value on the GUI and choose suitable equalizer mode.
Insertion_Loss_Eq_mode.png

Thanks & regards
Leo


------------------------------------------------------------------------------------------------

Don’t forget to reply, kudo, and accept as solution.

If starting with Versal take a look at our Versal Design Process Hub and our
Versal Blogs

------------------------------------------------------------------------------------------------
0 Kudos
joe306
Scholar
Scholar
708 Views
Registered: ‎12-07-2018

Hello, thank you for responding to my message. I have not made any changes to the Transceiver settings:

Aurora Transceiver SettingsAurora Transceiver Settings

I would not know where to begin adjusting things settings.

Concerning the mmcm_not_locked, that signal is only available when you select Include Shared Logic in example design:

Aurora2.jpg

When you select this setting it changes many input and output ports:

Aurora3.jpg

When the IP is updated I look the user_clk_out port is missing. I used that clock to attach to the m_axi_mm2s_aclk and m_axi_s2mm_aclk shown in yellow in the above graphic. The pin property of tx_out_clk shows a frequency of 97.65625Mhz. The user_clk_out was 48.828Mhz which is the max range on the init_clk. So I need some guidance on how to generate a clock for the AXI_DMA IP.

 

Last, the inputs refclk1_in, user_clk and sync_clk inputs what should be the value of user_clk? I see the sync_clk should be twice the user_clk.

 

Thank you very much for all your help.

Joe

0 Kudos
joe306
Scholar
Scholar
659 Views
Registered: ‎12-07-2018

Hello, can you please help me connect up the refclk1_in, sync_clk and user clk?

Thank you
karnanl
Xilinx Employee
Xilinx Employee
645 Views
Registered: ‎03-30-2016

Hello Joe @joe306 

I would suggest to generate Aurora 64B66B Example design from your XCI.
You will see something like :
    sync_clk and user_clk are connected to tx_out_clk ( using BUFG_GT ).
    refclk1_in is a reference clock for your transceivers, so you need to connect it to IBUFDS_GTE4

Aurora_64b66b_clock_connectivity_for_joe.png

Hope this helps.

Regards
Leo


------------------------------------------------------------------------------------------------

Don’t forget to reply, kudo, and accept as solution.

If starting with Versal take a look at our Versal Design Process Hub and our
Versal Blogs

------------------------------------------------------------------------------------------------
joe306
Scholar
Scholar
632 Views
Registered: ‎12-07-2018

Leo,

Thank you for responding to my message. I did create an example and it is very helpful:

I'm making some progress now. I'm running the Aurora at 1.25Gbps so my clock frequencies have changes and that causes conflicts with other things.

Will keep you informed.

 

Thank you very much,

Joe

joe306
Scholar
Scholar
576 Views
Registered: ‎12-07-2018

Leo,

HI, correct me if I'm wrong here, but if I'm running in Include Shared Logic then I only need to be concerned with:
If MMCM is used to generate a stable clock (init_clk), pma_init needs to be applied to
the Aurora 64B/66B core until MMCM lock is established. This ensures that the core remains
in a known state before a stable clock is available for the core.
From page 65.

If I'm running in Include Shared Logic in example design then I get the ports sync_clk, user_clk and tx_out_clk and the mmcm_not_locked port. In this case I need to use the mmcm_not_locked port.

It would appear to me that because I am running in Shared Logic and init_clk is from a MMCM then I only need to follow the documentation: If MMCM is used to generate a stable clock (init_clk), pma_init needs to be applied to
the Aurora 64B/66B core until MMCM lock is established. This ensures that the core remains
in a known state before a stable clock is available for the core. I tried this on the ZCU106 board and the loopback passed the test.

I don't think I have to run in "Include Shared Logic with example design"

Comments please?

Respectfully,
Joe
karnanl
Xilinx Employee
Xilinx Employee
551 Views
Registered: ‎03-30-2016

Hello Joe @joe306 

>If I'm running in Include Shared Logic in example design then I get the ports sync_clk, user_clk and tx_out_clk and the mmcm_not_locked port. In this case I need to use the mmcm_not_locked port.

>It would appear to me that because I am running in Shared Logic and init_clk is from a MMCM then I only need to follow the documentation: If MMCM is used to generate a stable clock (init_clk), pma_init needs to be applied to
the Aurora 64B/66B core until MMCM lock is established. This ensures that the core remains
in a known state before a stable clock is available for the core.

I believe your understanding are correct. (both of them)

>I don't think I have to run in "Include Shared Logic with example design"

Yes, this also correct.


This "Shared Logic" configuration is also applicable to most Xilinx IPs. So,

(a) If you only implement a single IP, you may prefer to configure your IP as "Include Shared Logic in core"
     Since using this setting, you will get an IP with more simple interface.

(b) If you want implement a multiple IPs in your design, you propably want to share clock resources between those IPs.
     If this is the case, you should configure one master IP as "Include Shared Logic in core"
     and please configure the rest of the IPs (slave IP) as "Include Shared Logic in example Design",
     and share the clock resources between those IPs.

Master_Slave_IP_con.png

Kind regards
Leo


------------------------------------------------------------------------------------------------

Don’t forget to reply, kudo, and accept as solution.

If starting with Versal take a look at our Versal Design Process Hub and our
Versal Blogs

------------------------------------------------------------------------------------------------
joe306
Scholar
Scholar
528 Views
Registered: ‎12-07-2018

Hello, thanks for responding to my message. I will use a Clock Wizard to generate init_cllk input signal to the IP. I will follow the the documentation on page 65:
If MMCM is used to generate a stable clock (init_clk), pma_init needs to be applied to the Aurora 64B/66B core until MMCM lock is established. This ensures that the core remains in a known state before a stable clock is available for the core.

I will also follow Figures 3-2 and 3-3.

The IP will be set for "Include Shared Logic"

Will let you know how things turn out.

Respectfully,
Joe