RFDC Evaluation Tool Failure at "VBG Trim" state

[epeach]

(Using ZCU111 with RFDC Evaluation Tool User Interface)

I have been using the RFDC Evaluation tool for a few weeks running the basic examples from the getting started guide (the one included in rdf0476-zcu111-rf-dc-eval-tool-2018-2.zip).

Today, I was trying to run the loopback example. I started the board and open the RFDC Evaluation Tool UI, and it looks like all the tiles failed to initialize.

The internal status says 'Failure at "VBG Trim" state'. I tried searching for this message and couldn't find any results. I think "vbg" might be referring to Band-Gap Voltage Reference though.

I was observing ESD Protection habits like using a blue static mat and being grounded before handling the board. As far as I know, there's nothing anyone on my team did to provoke a change.

Can someone at Xilinx help me get my board working again?

[epeach]

It never fails...

I posted my question after about 4 days of fiddling around trying to get it to work with no luck.

Sure enough, immediately after submitting my question, I try one more time and the board seems to have fixed itself. (╯°□°）╯︵ ┻━┻

The improvement came after some combination of hitting the 'Shutdown' and 'Reset' buttons on each tile.

[klumsde]

Hi @epeach 

Sorry this wasn't picked up... 

It seems odd that they all would stop at this step.

I'd expect this could be power supply related. The voltage reference of the converter is generated internally so a possible cause of the band gap trim failing is the power supply misbehaving. 

If this happens again, I'd suggest taking a look at the power advantage tool that comes with the design. It could point to the root of the problem. 

[epeach]

@klumsde , it happened again. This time, I'm unable to find a combination of (RESET, SHUTDOWN, START UP) that fixes the problem. I took screenshots of the PAT tool when it was still working and now while it is not. I can see that there is like a 10W difference in power usage but that could be for a lot of reasons if the design isn't starting up properly.

Working:

Not Working:

Now what?

[klumsde]

I agree the lower power consumption could be to do with the fact that the tiles are not started.

Can you tell me what step is vbg_trim is? There should be a number next to it in the GUI. I'll have to check what could really cause this here. 

Keith

[klumsde]

Also what is the state of the Power LEDs on the board?

[klumsde]

Hi 

I'm looking at ADC and DAC Avccaux. 

Both rails should be 1.8V 

In the non working case these rails are powered off. Both read only ~90mV

This is the reason the bandgap trim won't complete. 

[klumsde]

It is hard to tell the root cause of this.

It could be that both regulators are dead or there is some sort of damage to the device.

You could check vccint_ams_pwrgood, this signal enables both regulators to come up.

You'd likely need to have the infineon power dongle to properly debug the regulator error flags. 

[epeach]

Hi @klumsde 

Today I tried starting up the board and the DAC tiles seem to be green, but the ADC tiles are getting errors.

VBG Trim state is step 2,
I'm also occasionally getting "Failure at VCM/Bias" state (step 3) when I try and reset the tile.

For Status LEDs, All the lights are on (Callouts 60, 61, 62 on UG1271 pg 12) are on.
DS1 is also green. DS2 is green. DS 3-5 are off.

The problem is very intermittent. Sometimes I power on the board, start the GUI app and everything works, other times, I need to reset the tiles, and other times nothing can be done.

I don't have any diagnostic tools apart from what came in the ZCU111 box. Is this something I can likely fix myself or should I coordinate with my team lead and try to send the board to Xilinx?

[klumsde]

If we had the PMBus dongle and if there is an infineon gui we could use to access to the regulator we could try to reprogram it or we may see what the problem is. 

If that doesn't work then it seems it is damaged.

If the board is under warranty there is a process in place to RMA it. 

[wayne.punchak]

Hello @klumsde,

@epeach has gone on a vacation, but I have access to the board in our lab.

We have access to what I think is the dongle you're referring to: Max Power Tool L002 from Maxim Integrated.

Is this dongle compatible with the RFSoC and could it be used for the debugging you recommend?

Thanks.

[klumsde]

Hi,
The PMIC vendor in this case is Infineon.
Voltage and current monitoring and control are available for the Infineon power system
controllers through the Infineon PowIRCenter graphical user interface. The PMBus interface
controllers and regulators are accessed through the 1x3 PMBus
connector J19, which is provided for use with the Infineon PowIRCenter USB cable (Infineon
part number USB005) and can be ordered from the Infineon website

[klumsde]

Hi 

Do you know of the clocks from the PLLs are on? would you mind trying this with a newer version of the tool?

Download the version aligned to 2019. 1

Another thing to think of is making your own bitstream in 2019.1 then program that. Maybe try a bit stream with just a single DAC. 

[klumsde]

Another potential thing to check is that you don't boot anything. 

Put it into JTAG mode and connect with the System Controller GUI. 

If the ADC/DAC AVCCAUX is not powered then this will identify if the issue is with the regulator or if there is a dependency on the Converters booting up. 

Keith 

[wayne.punchak]

Hi @klumsde,

Please see my responses to your questions below in orange. I need some more information to try some of the debugging techniques you suggested.

The PMIC vendor in this case is Infineon.
Voltage and current monitoring and control are available for the Infineon power system
controllers through the Infineon PowIRCenter graphical user interface. The PMBus interface
controllers and regulators are accessed through the 1x3 PMBus
connector J19, which is provided for use with the Infineon PowIRCenter USB cable (Infineon
part number USB005) and can be ordered from the Infineon website

Your other debugging options seem more appropriate before we consider going out and buying more hardware.

Do you know of the clocks from the PLLs are on?

Where can I view this information?

would you mind trying this with a newer version of the tool?Download the version aligned to 2019. 1

What tool are you referring to? Where could I go to download the version of the referenced tool?

Another thing to think of is making your own bitstream in 2019.1 then program that. Maybe try a bit stream with just a single DAC. 

Does Xilinx have a Vivado design that I can build and flash to the board?

[wayne.punchak]

Another potential thing to check is that you don't boot anything. 

Put it into JTAG mode and connect with the System Controller GUI. 

If the ADC/DAC AVCCAUX is not powered then this will identify if the issue is with the regulator or if there is a dependency on the Converters booting up. 

@klumsde , @zhendon :

I've connected the System Controller GUI to our RFSOC (in JTAG boot mode), below are the results:

ClocksVoltagesPower

[wayne.punchak]

Afterwards, I also ran the board through the built-in self test. All tests passed.

[klumsde]

Hi @wayne.punchak 

In you image from the SCUI the ADC and DAC AVCCAUX are still unpowered. 

These supply rails come from LDOs. 

In the past we had an issue with these LDOs whenever the ADC and DAC clocks were not running during the power ramp. You would get a reverse current to this LDO which would cause it to alarm and stop. 

This should have been fixed in 2018.3 because we changed the start up state machine to eliminate this condition. 

So one thing to try is to replace the Evaluation design GUI and Boot images with the most up to date version

https://www.xilinx.com/support/documentation-navigation/see-all-versions.html?xlnxproducttypes=Boards%20and%20Kits&xlnxdocumentid=UG1287

Try to boot the 8x8 NON MTS design. If the issue is like the one descibed then this might make the problem go away. 

Alternatively try this out: 

Program the RF PLLs onthe board via the SCUI 

Pick the clock files shown here from the SCUI folder.

use the highlighted files to program the RF PLLs in the board.

Check that the LEDs shown on the edge of the board are lit, this will indicate locked. 

After this step I would think about trying the RF Analyzer bitstream then. you could download the Analyzer tool from here: https://www.xilinx.com/products/silicon-devices/soc/rfsoc.html#resources

It comes with a zu28DR bitstream that you could use. 

See if that gets it to come back to life.

Since a power cycle of the board and putting it into JTAG mode without booting anything shows ADC/DAC AVCCAUX rails down, I don't really hold out much hope of the board not being damaged. 

Try either of those steps above and let me know how you get on. 

Regards, 

Keith 

[wayne.punchak]

Hi @klumsde,

As you recommended, I booted up the 2019.1 boot image for the 8 x 8 NON MTS design. I'm now seeing power on the ADC / DAC AVCCAUX and have attached a screenshot below:

ZCU111 - Power with 2019.1 boot image

Does this mean that there's a bug in the Xilinx 2018.3 boot image / software?

[klumsde]

Hi @wayne.punchak 

I never observed this with that version of the tool. 

It could be some funny in the clock programming that is not easy to see. 

Anyway, it is booting reliably with the latest version is that right?

Keith 

[klumsde]

Hi,

Any update here?

Has the problem gone away?

[wayne.punchak]

Hi @klumsde,

I've booted it up 5 times in succession and have not seen the "VBG Trim" state failure...

When @epeach is back, I'll have him test it more thoroughly.

I'm hesitant to call this a fix because it was an intermittent problem. Additionally, our Vivado license is limited to 2018.3, the boot files you recommended were from 2019.1.

Does this mean that our team is going to see this error again once we start using the ADCs/DACs in our own custom designs created with the 2018.3 version of Vivado?

Thank you for your help.

[epeach]

Hi @klumsde , @wayne.punchak ,

I tried having another look using the new design, and found that I'm no longer getting the "VBG Trim" state failure, but there are a few other errors preventing me from completing the loopback example using the GUI.

---

@wayne.punchak wrote:

Does this mean that our team is going to see this error again once we start using the ADCs/DACs in our own custom designs created with the 2018.3 version of Vivado?

---

---

---

^ That's the real determinant for how we will proceed.

Steps:

Setup

• Installed the 2019.1 version of the RFDC Evaluation Tool UI.
• Using the 8x8 NON MTS design from the 2019.1 UG1287 design files.
• Set up as usual with the IP Addresses configured correctly on the board and computer.

Start

• Power on board and start GUI.
  • GUI connects to board. I get message
    ERROR: GetBitstream CMD_UNDEF.
  • I click continue and all the tiles turn green.

Try running the Loopback example instructions on on page 40 of 'ZCU111 RFSoC RF Data Converter Evaluation Tool -- Getting Started Guide (2018.2)'.

1. Set PLL Frequency of DAC Tile 1 to 6389.76MHz.
   1. I get warnings:
      ERROR: Get FabClkOutDiv CMD_UNDEF
      ERROR: Get MTSEnable CMD_UNDEF
2. Set T1/DAC0 crossbar to I/Q Mode.
3. Set up DAC0:
   • Try:
     • DAC 0 Frequency to 1500MHz.
     • Nyquist Zone to 1
     • Interpolation to 8x
     • Click 'Apply'.
   • I get error message:
     ERROR: SetFabClkOutDiv EXECUTE
     metal: warning:
     Tile_Id 1 Block_Id 1 has a different ratio 4.00000 between Fout and Fin, previous rati 1 will be used, try changing Decimation or Interpolation or IQ mode.
4. Ignore the error, click 'OK' and keep going and generate waveforms.
5. Setup the ADC Tile 1 ADC 01.
   • Try:
     • Calibration Mode 2
     • Nyquist Zone 1
     • Click Apply
   • Got Error:
     ERROR: SetCalibrationMode EXECUTE
     metal: error:
     Invalid Event Source, Event Source is not supported in 4GSPS ADC XRFdc_MixerRangeCheck.
   • And also, the field to set the mixer frequency is greyed out.

I assume these warnings are just from using a different design that isn't 100% compatible with the evaluation tool UI. Which design out of the ones listed in the 2019.1 'images' folder is most similar to the default 'rev-B' one that we got in the 2018.2 edition? Maybe I'll try that one next.

Thanks again,
Eric

[zhendon]

Hello @epeach 

Try copy all the files in images folder of 2019.1 trd design to you SD card can solve your problem.

[wayne.punchak]

@zhendon, could you please elaborate? Why would the 2019.1 trd design solve the issue we're seeing? Is there a bug in 2018.3?

[klumsde]

Hi @wayne.punchak @epeach @zhendon

Let's be clear here. 

The things you are seeing now are to do with added features the gui has to identify the design running on the board. 

I would say the best image to use is the 8x8 non MTS version... 

Keith

[zhendon]

---

@wayne.punchak  已写:

@zhendon, could you please elaborate? Why would the 2019.1 trd design solve the issue we're seeing? Is there a bug in 2018.3?

---

Hello Wayne,

Sorry for the misunderstanding. I meat you might need to copy all the folders and files in the "image" folder to the SD card. Not just the design you wanted to run.

[srjagarapu]

@zhendon
Copying all the files solved our ERROR: GetBitstream CMD_UNDEF.
Thanks
Srinivas