cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted
Observer
Observer
1,519 Views
Registered: ‎05-24-2020

JESD loses sync

Jump to solution

Setup - ZCU102 connected to ADS54J66EVM on HPC1
(ADC) ADS54J66 parameters - LMFS = 4421, K = 32, 307.2MSPS, Line rate = 6.144Gbps, Sysref = 2.4MHz, Mode 8 (bypass DDC)

Xilinx JESD IP parameters - 

GTHE4, Starting location = X0Y8, Static linerate = 6.144Gbps, PLL type = CPLL , Master channel = 1, 

RefClk = 153.6MHz, Glbl clk= 153.6MHz, LMFC buffer size = 1024, Sysref always off, Scrambling off, F = 2, K = 32, Default SYSREF Required on Re-Sync = SYSREF required.

About 2 - 3 hours after establishing sync, the link loses sync.

losesync_may18.JPG

 

 

https://www.xilinx.com/support/answers/66921.html

According to Xilinx AR 66921 -
"Once in SYNC, there are 3 main reasons a system may fall out of sync (requesting a resync):
CGS is lost on any lane
An incorrect transition from 0xBC to the start of ILA is detected
Misalignment in the received data is detected. This means alignment codes in the data are detected at unexpected positions.
A resync will be triggered when 8 successive multiframe alignment characters are detected in unexpected places (not at the end of a multiframe). "


Here is the sequence I follow - 
1) Program FPGA
2) Hold JESD core in reset
3) Program ADC registers and turn on Ref and Glbl clk.
4) Clear JESD core reset (after clearing reset, status register 0x03C is 0x00003333) (status register "Debug Status" page 28 of PG066)
5) Trigger Sysref (after triggering, status register 0x03C is 0x0000eeee indicating sync was achieved)

After 2-3 hours, the link loses sync and I checked JESD link error count registers (0x824) (page 39 of PG066) and they were all zero.
When JESD link loses sync, I issue a reset to the JESD core, clear it and trigger sysref, but that does not help. The status register 0x03C is 0x00000000 after clearing reset and even after triggering sysref. At this point I have to re-program the FPGA and perform the above steps to re-establish sync. I want to avoid reprogramming the FPGA every time the link loses sync. What are the steps I can take to ensure the link does not fall of sync ? Do you have any recommendations to resolve this ? Thank you



0 Kudos
1 Solution

Accepted Solutions
Highlighted
Observer
Observer
688 Views
Registered: ‎05-24-2020

@roym 

Thank you for your detailed informative response. I learned new concepts about using Xilinx GTs and would benefit from it in my firmware development. I appreciate your time and help.

@nathanx 
Thank you for your providing helpful guidelines and suggestions. They helped me get more knowledge of working with the JESD core.

The solution - 
I was using a hardware evaluation license and as detailed by @barriet in his response to my post https://forums.xilinx.com/t5/Serial-Transceivers/JESD204-GTH-transceiver-location-on-ZCU102/m-p/1130560 , the JESD core, once in sync, will remain in sync for a specific time duration depending on the configuration. That was essentially the reason why I was repeatedly observing rx_sync going low and none of the troubleshooting steps were helping.

I have purchased a PROJECT license for the JESD core and at the time of writing this post, the link has been in sync for over 24hrs.

Thank you for your helpful suggestions. I appreciate it  

View solution in original post

0 Kudos
21 Replies
Highlighted
Voyager
Voyager
1,497 Views
Registered: ‎08-02-2019

Hi @raj ,

I don't know, whether you have enough experience about handling JESD, it is an advance topic.

And I don't know, whether you are using a Reference Design as base or not.

If I were you, I'd start this kind of task by utilizing a similar Reference Design, because most of the time, solving this kind of problem/ to bring up this kind of design takes more time.

If you agree with me, I can recommend to you a similar reference design link to you.

It is a Texas Ins. ADC as yours and uses ZCU105 Ultrascale as yours and utilizes JESD.  

It has ILA results of ADC & JESD Ramp data output communication e.t.c.

You need to adjust speed rates, lane counts, and change target chip as yours.

Saban

 

<--- If reply is helpful, please feel free to give Kudos, and close if it answers your question --->
Highlighted
Observer
Observer
1,413 Views
Registered: ‎05-24-2020

@sabankocalThank you for your response and link to the reference design. I started with UltraScale JESD demo design (see attached pdf) as base and developed custom application based on my understanding of the reference design. So far, I have been able to establish sync with the ADC and capture raw sample data.

The issue I am trying to figure out is the JESD link stability. I am testing the ADC for its long term continuous operation with ZCU102 without losing sync. During testing, I have observed every time the link establishes sync, it goes out of sync 2-3 hours later. The link has never been able to maintain sync for more than 3 hours and every time it loses sync, the link error counters are 0. Triggering sysref again at this point does not re-establish sync and I have to re-program the FPGA. Do you have recommendation on how to debug and resolve this issue ? 

Is the link supposed to remain stable and in sync as long as the power supply and clock are stable ? Are there other factors that affect the link ?

0 Kudos
Highlighted
Voyager
Voyager
1,408 Views
Registered: ‎08-02-2019

Hi @raj ,

Very strange issue. I don't have any idea about resolving problem, but If I were you

  • I'd create an System ILA core to debug all of related signals and AXI Interfaces.
  • Then I set a trigger on ILA as Falling or raising edge of this link state. Maybe position can be near to end of sample count (900/1024) to see what is going on before link state going down.

Saban

   

<--- If reply is helpful, please feel free to give Kudos, and close if it answers your question --->
Highlighted
Observer
Observer
1,404 Views
Registered: ‎05-24-2020

@sabankocal @roym I set a trigger on rx_sync signal and the attached screenshot in my first post is the sync signal going down. I observed there were no disparity errors before going out of sync, there was no incorrect transition from 0xBC to the start of ILA and there was no misalignment character in the received data because I have turned off sending frame align characters in the ADC settings. I am not sure if this is a known issue with ZCU102 boards. How do I check the power to the FPGA is stable ?

0 Kudos
Highlighted
Voyager
Voyager
1,347 Views
Registered: ‎08-02-2019

Hi @raj ,

What about System Monitor and XADC to monitor system. As you mentioned about power, I think high temprature, maybe cooling can be a problem.

Saban

 

<--- If reply is helpful, please feel free to give Kudos, and close if it answers your question --->
Highlighted
Observer
Observer
1,271 Views
Registered: ‎05-24-2020

@sabankocal @avrumw @justint @cbalakr @gguasti @kvasantr  @ddn  I monitored the temperature and voltages in SystemMonitor and all the numbers look normal. There was no unusual behavior. 

0 Kudos
Highlighted
Moderator
Moderator
1,181 Views
Registered: ‎08-01-2007

1) Firstly check if the rx_sync loss is due to GT 8b10b errors(not in table errors-gt*_rxdisperr, disparity errors - gt*_rxnotintable)

2) What your clock scheme? Are you following the clock scheme on PG066?

3) Dump all the registers when rx sync loss happens.

4) sysref alwasy set to on.

Please accept it as solution or give kudos if it answers your question.

Highlighted
Observer
Observer
1,112 Views
Registered: ‎05-24-2020

Hi @nathanx

Thank you for your suggestion. 

1) Looks like rx_sync loss was not due to rxdisperr or rxnotintable errors. On observing chipscope data, these registers were 0 when sync was lost. 

rx_sync goes lowrx_sync goes low



2) The JESD rx module is "Include Shared Logic in Core". F=2, K=32, LMFC buffer size=1024, Sample sysref on negative edge, CPLL. 

JESD204_rxJESD204_rx

I believe the core internally instantiates IBUFDS, BUFG_GT and connects gt_powergood to CE pin of BUFG_GT.

3)  I interpret "dump all registers" as write desired values to AXI registers again. After losing sync, I wrote 0x00000002 to 0x004 register to reset the core and then 0x00000000 to clear the reset. I then wrote desired values to other registers (0x008, 0x00C, 0x010, 0x018, 0x020, 0x024, 0x028, 0x02C, 0x030, 0x034). After this I issued a reset to the rx_reset pin by sending a 1 and cleared it by sending 0. I observed 0xBCBC characters on the GT lanes. 
 GT lane data after losing syncGT lane data after losing sync

 

But the "Debug status" register 0x03C read 0x00000000 indicating the core did not acknowledge 0xBCBC characters. Based on my experience, if the core acknowledges 0xBCBC data on GT lanes, the "Debug status" register 0x03C reads 0x00003333. But after losing sync, this register reads 0x00000000 no matter how many times the core is reset and all the AXI registers are dumped. Am I missing something ?

4) Sysref is set to always on.

Do you have any recommendations on debugging this ? 


I have eye diagrams for all 4 lanes. 

Lane 0Lane 0Lane 1Lane 1Lane 2Lane 2Lane 3Lane 3

0 Kudos
Highlighted
Moderator
Moderator
1,066 Views
Registered: ‎08-01-2007

"Dump all register" means reading all the IP registers? Can you read all the registers and post all the register values when sync is lost?

0 Kudos
Highlighted
Observer
Observer
1,047 Views
Registered: ‎05-24-2020

Hi @nathanx 

This is the AXI register dump for the RX core

BEFORE                 AFTER

07020000   0x000   07020000
00000000   0x004   00000000
00000001   0x008   00000001
00000000   0x00C  00000000
00000001   0x010   00000001

00000000   0x018   00000000
00000000   0x01C  00000000
00000001   0x020   00000001
0000001f    0x024   0000001f
0000000f    0x028   0000000f
00000001   0x02C  00000001
0000002c   0x030   0000002c
00000001   0x034   00000001
00010001   0x038   00010000
0000eeee   0x03C  00000000

LANE 0

0x800 00000000
0x804 00000000
0x808 00000000
0x80C 00000000
0x810 00000000
0x814 00000000
0x818 00000000
0x81C 00000000
0x820 00000000
0x824 00000000
0x828 00000000
0x82C 00000000
0x830 00000000

LANE1

0x840 00000000
0x844 00000000
0x848 00000000
0x84C 00000000
0x850 00000000
0x854 00000000
0x858 00000000
0x85C 00000000
0x860 00000000
0x864 00000000
0x868 00000000
0x86C 00000000
0x870 00000000

LANE 2

0x880 00000000
0x884 00000000
0x888 00000000
0x88C 00000000
0x890 00000000
0x894 00000000
0x898 00000000
0x89C 00000000
0x8A0 00000000
0x8A4 00000000
0x8A8 00000000
0x8AC 00000000
0x8B0 00000000

LANE 3

0x8C0 00000000
0x8C4 00000000
0x8C8 00000000
0x8CC 00000000
0x8D0 00000000
0x8D4 00000000
0x8D8 00000000
0x8DC 00000000
0x8E0 00000000
0x8E4 00000000
0x8E8 00000000
0x8EC 00000000
0x8F0 00000000 

 

0 Kudos
Highlighted
Moderator
Moderator
1,019 Views
Registered: ‎08-01-2007

It does not make sense. The register values do not match the data we already saw on gtN_rxdata[31:0]. The value of address "0x03C" shows Lane does not receive K28.5's, but from the screenshot you shared before, I can see 0xbcbc on gtN_rxdata[31:0]. That said, the value of address "0x03C" contradicts the 0xbcbc data on gtN_rxdata[31:0] we saw.

Are you dumping register and capture gtN_rxdata from the same link? I guess you may make a mistake. Make sure you are debugging the same link.

You also said the link operated for a few hours, after that, you saw the rx_sync loss. This shows to me the parameter settings between TX(ADC) and RX(Xilinx JESD204 IP core) should match, because the link was up and running for a few hours.

It looks to me some errors happens, which causes link loss, it might be due to clock sent to FPGA or ADC not stable, or the power suppply not very clean, etc.

 

Please accept it as solution if it answers your question.

0 Kudos
Highlighted
Observer
Observer
1,000 Views
Registered: ‎05-24-2020

@nathanx 

Are you dumping register and capture gtN_rxdata from the same link? I guess you may make a mistake. Make sure you are debugging the same link.


I made sure I'm debugging the same link. I'm inclining towards some bug in the JESD core which is causing it to not detect 0xBCBC characters after sync loss. I have to reprogram the FPGA for the link to establish sync indicating the JESD core may be stuck. Or can it be a defect in ZCU102 board ?

It looks to me some errors happens, which causes link loss, it might be due to clock sent to FPGA or ADC not stable, or the power supply not very clean, etc..

Do you have any recommendations on how to debug the stability of clock and power supply?

0 Kudos
Highlighted
Moderator
Moderator
981 Views
Registered: ‎08-01-2007

I believe you must make something wrong, as the info you provided does not make sense. I would recommend you to look at your design again, alternatively you can build another JESD link on your board to verify it, which is just a loopback test, with a JESD204 IP TX and JESD204 RX included in FPGA.

The JESD204B is a mature and robust IP core, you can take a look at IP change log to check the IP changes(including bug fix). Staring from Vivado 2018.2, the IP has no bug fix.

 

Please accept it as solution if it answers your question.

 

 

 

0 Kudos
Highlighted
Observer
Observer
843 Views
Registered: ‎05-24-2020

@nathanx @roym 

Thank you for your suggestion. I setup a loopback test like below - 

Vivado 2017.3
ZCU102 external loopback with XM107ZCU102 external loopback with XM107

JESD RX -  Line rate = 6.25Gbps, LMFS = 4421, K=32, RefClk = 156.25MHz, CoreClk = 156.25MHz, Subclass 0, CPLL


JESD TX -  Line rate = 6.25Gbps, LMFS = 4421, K=32, RefClk = 156.25MHz, CoreClk = 156.25MHz, Subclass 0, QPLL0

Refclk --> IBUFDS_GTE4 --> BUFG_GT --> tx and rx coreclk

ZCU102 external loopback with XM107 vivado block diagramZCU102 external loopback with XM107 vivado block diagram

ZCU102 external loopback with XM107 JESDPHY blockZCU102 external loopback with XM107 JESDPHY block

The Initialization sequence is - 

1) Program FPGA

2) Send 1 to rx_reset, tx_reset pin of the JESD RX module and rx_sys_reset, tx_sys_reset of JESD PHY (highlighted in above photo). JESD_Reset_axi module is custom AXI module to reset JESD module and monitor JESD signals. 

3) Send 0 to tx_reset and tx_sys_reset pin of JESD TX and JESD PHY module.

4) Write 0x2 to JESD RX and JESD TX AXI register 0x04 (fixed reset) to hold the JESD core in reset.

5) Send 0 to tx_reset and tx_sys_reset pin of JESD TX and JESD PHY module.

6) Write 0x0 to  JESD TX AXI register 0x04  to clear JESD core reset.

7) gtN_txdata lanes start sending 0xBCBC characters.

Send 0 to rx_reset and rx_sys_reset pin of JESD RX and JESD PHY module.

9) Write 0x0 to  JESD RX AXI register 0x04  to clear JESD core reset.

10) rx_sync  goes high indicating sync established.

No need to trigger sysref since JESD RX and TX are setup to be in Subclass 0 mode.

The link established sync and after about 2.5hrs, rx_sync went low. I observed a signal named "resync_trig" inside JESD RX module goes high one clock cycle before rx_sync goes low and not sure what drives it high.

ZCU102 external loopback with XM107 sync lossZCU102 external loopback with XM107 sync loss

I had setup a trigger such that it will trigger whenever gtN_rxdisperr[3:0] or gtN_rxnotintable[3:0] are not equal to 0 or when rx_sync goes low. The trigger went off on rx_sync going low.

The ILA trigger did not trigger on gtN_rxdisperr[3:0] or gtN_rxnotintable[3:0]  not equal to 0.

 

The AXI register dump before and after Sync is below -

JESD RX

BEFORE                   AFTER

07020000   0x000   07020000
00000000   0x004   00000000
00000001   0x008   00000001
00000000   0x00C  00000000
00000001   0x010   00000001

00000000   0x018    00000000
00000000   0x01C   00000000
00000001   0x020    00000001
0000001f    0x024    0000001f
0000000f    0x028    0000000f
00000000   0x02C   00000000
00000000   0x030    00000000
00000001   0x034    00000001
00000001   0x038    00000000
0000eeee   0x03C   00000000


LANE 0
00000100   0x800   00000000
00000001   0x804   00000000
0000001f    0x808   00000000
03000000   0x80C   00000000
010f0d03    0x810   00000000
00000000   0x814   00000000
00440000   0x818   00000000
00000000   0x81C   00000000
00000000   0x820   00000000
00000000   0x824   00000000
00000000   0x828   00000000
00000000   0x82C   00000000
00000000   0x830   00000000

LANE1
00000100   0x840   00000000
00000001   0x844   00000000
0000001f    0x848   00000000
03010000   0x84C   00000000
010f0d03    0x850   00000000
00000000   0x854   00000000
00450000   0x858   00000000
00000000   0x85C   00000000
00000000   0x860   00000000
00000000   0x864   00000000
00000000   0x868   00000000
00000000   0x86C   00000000
00000004   0x870   00000000

LANE 2
00000100   0x880   00000000
00000001   0x884   00000000
0000001f    0x888   00000000
03020000   0x88C   00000000
010f0d03    0x890   00000000
00000000   0x894   00000000
00460000   0x898   00000000
00000000   0x89C   00000000
00000000   0x8A0   00000000
00000000   0x8A4   00000000
00000000   0x8A8   00000000
00000000   0x8AC   00000000
00000004   0x8B0   00000000

LANE 3
00000100   0x8C0 00000000
00000001   0x8C4 00000000
0000001f    0x8C8 00000000
03030000   0x8CC 00000000
010f0d03    0x8D0 00000000
00000000   0x8D4 00000000
00470000   0x8D8 00000000
00000000   0x8DC 00000000
00000000   0x8E0 00000000
00000000   0x8E4 00000000
00000000   0x8E8 00000000
00000000   0x8EC 00000000
00000004   0x8F0 00000000

 

JESD TX

BEFORE                   AFTER

07020000   0x000   07020000
00000000   0x004   00000000
00000001   0x008   00000001
00000000   0x00C   00000000
00000001   0x010   00000001
00000003   0x014   00000003
00000000   0x018   00000000

00000001   0x020   00000001
0000001f    0x024   0000001f
0000000f    0x028   0000000f
00000000   0x02C   00000000


00000001   0x038   00000000

 

00000000   0x80C   00000000
010f0d03    0x810   010f0d03
00000000   0x814   00000000
00000000   0x818   00000000
00000000   0x81C  00000000

After rx_sync goes low, I tried resetting the JESD PHY, RX and TX block but the RX AXI "Debug Status" register 0x03C reads 0x00000000. This indicates the JESD RX module is not acknowledging the 0xBCBC characters, but however, from the screenshot above, I can observe 0xBCBC data on the gtN_rxdata lanes.

How can I debug this more ?

 

 

 

0 Kudos
Highlighted
Observer
Observer
835 Views
Registered: ‎05-24-2020

@nathanx @roym 

To rule out the possibility of  ZCU102 board being faulty, I also tested the same configuration on a KCU105 board and the results were same. The JESD RX link loses sync after about 2hr5mins to 2hr25mins. It does not stay in sync for more than 2.5hrs. 

I suspected three causes for this issue - 

1) Power 

To ensure stable power supply, I connected the board to an external DC power supply.

2) Clock

Perform IBERT Near-end PMA loopback test on the transceiver using PRBS 31bit pattern. Refclk source to the IBERT core  was once U56 Si570 (156.25MHz, 6.25Gbps) and the other time using LMK04828 (163.65MHz, 6.546Gbps). I let this test run for over 3 days in both cases (Si570 and LMK04828) and there were no errors observed. I performed this test on both ZCU102 and KCU105 boards. Please see below photo for KCU105 test. 

KCU105 Near-end PMA test, refclk 163.65MHz from LMK04828. PRBS 31bitKCU105 Near-end PMA test, refclk 163.65MHz from LMK04828. PRBS 31bit

3) Signal Integrity

I have tested two different ADS54J66EVM boards and two FPGA boards (ZCU102 and KCU105). I have also tested external loopback test using XM107 loopback card on both ZCU102 and KCU105 boards.

Even after this, the link is not stable for more than 2.5hrs.  Is there anything I am missing or doing incorrectly ?


 

0 Kudos
Highlighted
Moderator
Moderator
822 Views
Registered: ‎07-30-2007

1. it looks to me like the lane on the bottom right does not have enough width in the Eye Diagram.  If you could add some preemphasis in the TI chip that would help center it better.

2.  If you could turn on scrambling I expect the channel would work better.  Has this been tried?  Can it be tried?  

3.  I suspect if you are using DFE you could be getting a bad adaptation over time.  You should see what happens if you put the equalization attributes in HOLD after running 10 minutes. See the attributes starting on about page 195 of UG576.   Does that increase your run time?

Do you which equalization you are using?  If not look in your design for the LPMEN setting?

 




----------------------------------------------------------------------------
Don't forget to reply, kudo, and accept as solution
----------------------------------------------------------------------------


Highlighted
Observer
Observer
816 Views
Registered: ‎05-24-2020

@roymThank you for your response. 

1) The current emphasis setting in the ADC is 0 dB. The available values are  0 dB, –1 dB, –2 dB, –4.1 dB, –6.2 dB, –8.2 dB, –11.5 dB. I am going to try all these values to get a better eye diagram. 

2) Currently, scrambling is off. I will turn it on on ADC and FPGA and monitor the results.

3) Equalization was set to "Auto" in the JESD PHY gui block. I read 0x608 register of the PHY module and it read 1 indicating LPM was enabled. If I set this to "Low Loss" in the PHY gui, it will select LPM and automatically select 14dB insertion loss. Should I change the insertion loss value ?

I have the JESD PHY module "Shared logic in core", transceiver debug ports enabled and AXI enabled. However I do not see the HOLD inputs/attributes available as an input. Do you have a suggestion on how to write to those ports/attributes ?

JESD PHY blockJESD PHY block

 

 

 

LPM hold inputLPM hold input

 

UG576 Page 201UG576 Page 201

0 Kudos
Highlighted
Moderator
Moderator
783 Views
Registered: ‎07-30-2007

I would start with the -4.1 dB setting.  One thing that could be happening is that the repeated BCBC could be bad for the adaptation of the equalization.  That is if long strings of the commas are sent.  Any data that repeats for long periods of time could be a potential problem.    In that case just set the holds when the repeating pattern starts and release them when it stops.  

Getting the ports out of the design will be tedious.  You will have to make an rtl version of the design with the GT in the example design.  Since the GT doesn't work with the IPI flow you would have to copy the rtl example design.  You would need to put a wrapper around the GT in the example design to make it into an IPI model that you could connect with the rest of the IP in your IPI based design.  

One thing you could try first is to do away with the repeating patterns on the transmit end if you can keep the TX transmission rate high and see if that makes the problem go away.  If it does then first try the easiest thing of turning on the scrambling, assuming that gets rid of the repeats.  Otherwise you'll have to expose the other ports to test this.

And yes it would be a good idea to make a design with the Insertion loss set to 16.  This will turn on DFE so you can see if the DFE gives you more margin in the eye scan.

 

 




----------------------------------------------------------------------------
Don't forget to reply, kudo, and accept as solution
----------------------------------------------------------------------------


Highlighted
Xilinx Employee
Xilinx Employee
771 Views
Registered: ‎05-01-2013

Does the IP have the full license?

You can run "report_ip_status" in Vivado tcl in your project to check all the IP cores' license status.

Highlighted
Moderator
Moderator
706 Views
Registered: ‎08-01-2007

Since the loopback test does not work. I would suggest to check the following:

1) What's the clock scheme? Are you following the clock scheme of Figure 3‐5: Artix-7, Kintex-7, Virtex-7 Subclass 0 Clocking on PG066?

2) Subclass 0 does not require sysref, but it looks to me you have used sysref.

3) Check all the signal connections bettween JESD TX, RX and PHY core, take a look at Figure 3‐20: Two Lane TX and Two Lane RX Sharing Transceivers or Figure 3‐21: Connections Required between JESD204 and JESD204_PHY on PG066.

Highlighted
Observer
Observer
689 Views
Registered: ‎05-24-2020

@roym 

Thank you for your detailed informative response. I learned new concepts about using Xilinx GTs and would benefit from it in my firmware development. I appreciate your time and help.

@nathanx 
Thank you for your providing helpful guidelines and suggestions. They helped me get more knowledge of working with the JESD core.

The solution - 
I was using a hardware evaluation license and as detailed by @barriet in his response to my post https://forums.xilinx.com/t5/Serial-Transceivers/JESD204-GTH-transceiver-location-on-ZCU102/m-p/1130560 , the JESD core, once in sync, will remain in sync for a specific time duration depending on the configuration. That was essentially the reason why I was repeatedly observing rx_sync going low and none of the troubleshooting steps were helping.

I have purchased a PROJECT license for the JESD core and at the time of writing this post, the link has been in sync for over 24hrs.

Thank you for your helpful suggestions. I appreciate it  

View solution in original post

0 Kudos