cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
jim_godfrey1
Visitor
Visitor
1,347 Views
Registered: ‎05-25-2021

Spartan 7 BRAM/FIFOs glitch until JTAG pod is connected

I see glitches on data running thru BRAM and FIFO cores.  Once I Auto Connect the Platform Cable USB II the glitches disappear, even when I physically disconnect the pod.  Powerup sequence and supplies look good.

0 Kudos
27 Replies
jim_godfrey1
Visitor
Visitor
1,280 Views
Registered: ‎05-25-2021

Problem Description:
bram_core core appears to glitch sporadically. When the
Platform Cable USB II (JTAG pod) is connected (using "Auto Connect"),
this behavior goes away for good, even if the pod is detached.

In order to isolate this issue, a simple design with just a
clock input (clock) and 3 test outputs was created. A toggle
signal (toggle) was created, and input into bram_core. The
bram_core address was fixed to address 0 and the write enable
set always active. No reset is needed since the initial
state of the toggle signal does not matter and bram_core does
not have a reset input. A delayed version of this toggle
signal (toggle_d1) is compared to the bram_core output (bram_out),
and an error signal (error) goes hi if they do not match.

Input:
clock

Outputs:
fpga_dbg[0]: error
fpga_dbg[1]: buffered version of bram_out
fpga_dbg[2]: buffered version of toggle_d1

Design was simulated using Vivado's "Run Simulation".
All Vivado 2020.2 (64-bit) defaults were used to generate the bitstream.
The build is clean (no errors or warnings), and timing is met.
..\bram_glitch.srcs\utils_1\imports\bram_glitch\write_cfgmem.tcl
is used to generate the .mcs file. This script expects the
design to be installed at C:\Local. Scope was set to trigger on
rising error, and the other two outputs were viewed. bram_core
was created with Block Memory Generator 8.4.

This was tested on our proprietary board with pin M19 PUDC_B
pulled to GND with a 0 Ohm resistor. INIT_B is held low until
all supplies are stable, then pulled hi to start boot from flash.
Board was checked and power sequencing was correct. All supplies
look clean and glitch free. Clock is 100MHz from a Abracon
ASEM1-100.000MHZ-LC oscillator and fanned out to also go

to EMCCLK. Both of these and all banks are powered by 3.3V.

0 Kudos
drjohnsmith
Teacher
Teacher
1,263 Views
Registered: ‎07-09-2009

Sorry , cant open zip on the tablet, so a guess, hope some one else can look in depth.

would it be true to say this is on your own board design or is it on many different board designs ?

it sounds to me off top of head as if its a power / reset glitch,

    is the BRAM associated circuit connected to a reset 

     have you looked with scope at the signals / power when you do the connect ,

  If anything its sounding like an earth problem, earthing through the PC 

        suggest you try a different PC and / or ook at one of the USB isolators out there, 

 

<== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
jim_godfrey1
Visitor
Visitor
1,255 Views
Registered: ‎05-25-2021

Yes, it's our design.

The BRAM does not have a reset input.  The only user logic is a toggle signal into the BRAM, and it's initial state does not matter.  See the README for a description of the design, long story short I don't think its a reset problem in the user design.

The problem appears when the pod is not connected.  There is only one GND connection and that's on the input power return.

I'm not discounting a possible H/W issue with our board, but it would be nice to see this design working on a known good board.  Unfortunately, we do not have any Xilinx eval boards in house, so I was hoping one of the Xilinx engineers could try it.

0 Kudos
avrumw
Expert
Expert
1,246 Views
Registered: ‎01-23-2009

Obviously we will reevaluate if someone is able to reproduce this problem on another board, but I agree, this is most likely a power integrity issue (or something else associated with this board); the BRAMs have their own power supply which, on some packages, are tied internally to VCCINT, and on others have separate VCCBRAM pins - a power corrupt on VCCBRAM might explain something like this (although the JTAG pod is still weird).

This cannot be an FPGA design flaw - it is way too basic for it to have gone undetected. The BRAMs are reliable - this has been demonstrated in countless designs. What you are describing is that the RAM fails to operate in its most basic operational mode unless the JTAG pod is connected (or was once connected) - again, countless designs have shown that this is not the case.

So it has to be something specific to your board...

Avrum

jim_godfrey1
Visitor
Visitor
1,231 Views
Registered: ‎05-25-2021

That might be.  I think the next step is to run this same simple design on a known good board.  Does any of the support group have a S7 board, preferably with the same device, that can give it a try?  You will also need a decent scope, since signals are toggling at 100MHz.

0 Kudos
pthakare
Moderator
Moderator
1,103 Views
Registered: ‎08-08-2017

Hi @jim_godfrey1 

As you might be knowing  In general, all setup and hold times of a block RAM must be met in order for the block RAM to function as defined in the User Guide.

Reference:https://www.xilinx.com/support/answers/42571.html

So wanted to check if you can share us with timing report to do the sanity check .I will also review your share achieve and check the timing.

As @avrumw mentioned , i suspect this is to be related to power intigrity issues and needs following details to review here

#1 Share the VCCINT,VCCBRAM,VCCAUX &VCCO(VCCO Banks of flash related pins) Scope shots (one below the another) when you didn't  connect JTAG Cable and when you connect JTAG Cable.

  Note : Do the above step when you boot from flash.

#2  If the clock driving BRAM coming from external source on clock capable IO, then share the Scopeshot of that clock when you didn't connect JTAG Cable and when you connect JTAG Cable.

   Note : Do the above step when you boot from flash.

#3 Share your Power distribution network for our reference.

#4 Share the configuration interface schematic of Flash and JTAG.

#5 What is the behavior when you program the FPAG using JTAG interface instead of loading the configuration data from flash.

-------------------------------------------------------------------------------------------------------------------------------
Reply if you have any queries, give kudos and accept as solution
-------------------------------------------------------------------------------------------------------------------------------
0 Kudos
jim_godfrey1
Visitor
Visitor
1,072 Views
Registered: ‎05-25-2021

This is a proprietary design and I only want this to be distributed to Xilinx.  Is it possible to setup this forum to achieve this goal?

0 Kudos
pthakare
Moderator
Moderator
1,039 Views
Registered: ‎08-08-2017

Hi @jim_godfrey1 

Do you have access to file the Service request ? Or you in contact with FAE (Avnet or Xilinx) so that they can create a SR for your ?

The SR is recommended here to debug this further and also involved power experts in discussion.

Please check , else i will send you the Xilinx FTP Ezmove link to share the requested information above 

-------------------------------------------------------------------------------------------------------------------------------
Reply if you have any queries, give kudos and accept as solution
-------------------------------------------------------------------------------------------------------------------------------
0 Kudos
jim_godfrey1
Visitor
Visitor
1,009 Views
Registered: ‎05-25-2021

I re-ran the test on a different board design, and still see the core glitch issue. 

This is not a BRAM issue, since I also see it if I insert a FIFO built with flops in the data path (and no BRAM anywhere in the build, see attached).  Timing is met fine, there is plenty of slack.  I put a galvanic isolator on the USB and still see the issue disappear when I do an “Auto Connect”, and stay good even if I physically disconnect the JTAG.  All FPGA rails sequence on correctly and look clean, and I don’t see spikes or anything when doing the “Auto Connect”.  The clock input looks clean and I don’t see spikes or anything when doing the “Auto Connect”.  I added a MMCM on the clock input, and this has no effect, the MMCM never goes out of lock.  If I boot from JTAG instead of flash, I don’t see the issue.  The only thing that’s connected to the board is a isolated supply, and earthing it has no effect.  I shared the PDN and JTAG sections of our design with our local FAE, but cannot post these on a public forum.  They do follow Xilinx recommendations and decoupling caps are on the side opposite the BGA.

Are there any Xilinx internal signals that I can monitor/control?  As far as I can tell, something is happening “behind the curtain” of Xilinx logic, such that the cores operate correctly once an “Auto Connect” is done.  I've heard mention of a “Global Write Enable”.  What’s that?  Can I monitor it?

0 Kudos
pthakare
Moderator
Moderator
1,005 Views
Registered: ‎08-08-2017

Hi @jim_godfrey1 

Thanks for the further explanation around this. Recently we observed similar issue and we are debugging from configuration perspective .

I have forwarded this to our configuration expert .

Just quick question , what is your configuration mode ?

Our config expert will assist on  to ensure that startup sequence completes all the way to step7.

The step6 is assert GWE ,Allowing the RAMs and flip fops to change state.

 

pthakare_0-1622644293214.png

 

 

 

 

-------------------------------------------------------------------------------------------------------------------------------
Reply if you have any queries, give kudos and accept as solution
-------------------------------------------------------------------------------------------------------------------------------
drjohnsmith
Teacher
Teacher
975 Views
Registered: ‎07-09-2009

wow

this is going to be interesting to follow

 

<== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
0 Kudos
jim_godfrey1
Visitor
Visitor
968 Views
Registered: ‎05-25-2021

Thank you !

See attached

mode.png
0 Kudos
jim_godfrey1
Visitor
Visitor
968 Views
Registered: ‎05-25-2021

I see DONE as expected at the end of boot from x4 SPI flash.

0 Kudos
drjohnsmith
Teacher
Teacher
942 Views
Registered: ‎07-09-2009

Just a suggestion,

   I'd normally put 10K pull ups on the M lines ( im not even certain they need them, they might have internal pull ups )

      and then only change the zero ohm resistors on the pull down as needed, 

           tends  to be easier to see whats happening and easier to change, 

sorry, its just  a view, 

   still looking forward to more details on this,

 

<== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
0 Kudos
jim_godfrey1
Visitor
Visitor
833 Views
Registered: ‎05-25-2021

@pthakare

When can I expect a reply?

 

0 Kudos
pthakare
Moderator
Moderator
784 Views
Registered: ‎08-08-2017

Hi @jim_godfrey1 

I had discussion with config expert and you can expect the reply today .

-------------------------------------------------------------------------------------------------------------------------------
Reply if you have any queries, give kudos and accept as solution
-------------------------------------------------------------------------------------------------------------------------------
seamusbleu
Explorer
Explorer
762 Views
Registered: ‎08-12-2008

This topic has caught my interest.  So I looked up "Global Write Enable" (GWE) in UG470 series 7 configuration guide.  It says "Global Write Enable (GWE). When asserted, GWE enables the CLB and the IOB flip-flops as well as other synchronous elements on the FPGA" and has a footnote that says:

"GWE is asserted synchronously to the configuration clock (CCLK) and has a significant skew across the part. Therefore, sequential
elements are not released synchronously to the user's system clock and timing violations can occur during startup. It is
recommended to reset the design after startup and/or apply some other synchronization technique."

Which is quite interesting and a little alarming.

You can look at the status of GWE in bit 6 of the CONFIG_STATUS register in Vivado (Lab).  Also look at the GWE_CYCLE bits 2:0 in the COR0 register just under the CONFIG_STATUS register to see in which startup phase GWE deasserts (though I think they mean asserts there? - see table 5-31).

seamusbleu_0-1622806161542.png

Given all this, I'm wondering if you have a clocking/reset issue that is sticking from startup.  In your simple design I don't think you specifically do anything for clocking or resets.  If you open up the implementation, does the synthesizer add a clock buffer for you?  I'm not sure how the above would affect you, but one idea would be to add an MMCM, and create a design wide reset that asserted until after the MMCM locks.  I'm curious to see what the Xilinx experts have to say.

Also, look at this forum post on GSR.

 

== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
Tags (1)
jim_godfrey1
Visitor
Visitor
745 Views
Registered: ‎05-25-2021

@pthakare I didn't receive anything yesterday.

0 Kudos
jim_godfrey1
Visitor
Visitor
743 Views
Registered: ‎05-25-2021

Thanks for the info.

As mentioned earlier "I added a MMCM on the clock input, and this has no effect, the MMCM never goes out of lock."  I used the lock to reset my logic.  The BRAM core has no reset.  I'll try to add one to see if that helps.

If I connect the JTAG pod, the data glitch disappears, so I can't use the pod to debug.

seamusbleu
Explorer
Explorer
738 Views
Registered: ‎08-12-2008

I missed that (the MMCM).  I think it's a good idea to add the reset to the BRAM to rule that out, though I doubt it will do anything as the documentation says the reset only affects the output latches, not the memory cells. 

I'll add that it sounds like you've been very thorough about this, so kudos to you too.  Did you take a look at the two GWE settings  in the STATUS and COR0 registers?  I'm curious what they say, though I don't expect anything to jump out, it would be good to document them, even if it does require the JTAG pod connection, which makes the issue disappear.

== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
0 Kudos
seamusbleu
Explorer
Explorer
736 Views
Registered: ‎08-12-2008

Another thought would be to rearrange your BRAM inference logic to change the BRAM mode.  Check in implementation - is it currently implemented as read-first, write-first or no-change?  By changing how the logic is written, you should be able to change that.  I'm curious if that makes an difference to the errors you are seeing (or make them go away).  You can go to Tools->Language_Templates->Synthesis Constructs->Example Modules->RAM->... to see examples.

== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
0 Kudos
jim_godfrey1
Visitor
Visitor
728 Views
Registered: ‎05-25-2021

It's not a BRAM problem.  I see the same thing if I use a FIFO, and implement it using flops (and I checked that there were 0 BRAM in the implementation report).

0 Kudos
jim_godfrey1
Visitor
Visitor
672 Views
Registered: ‎05-25-2021

See reg values attached.

cfg_regs_w_pod.png
cor0_regs_w_pod.png
0 Kudos
seamusbleu
Explorer
Explorer
657 Views
Registered: ‎08-12-2008

Everything looks right from what you've said and done.  I'm curious as to what Xilinx config expert has to say - if they've seen something like this before. 

Another shot in the dark test.  In UG470, it says the following (below).  I wonder what would happen if you enable the LOCK_cycle or Match_cycle options, currently you have them at the default no wait settings.  Also, it's probably worth a shot changing your configuration clock select/rate.  I think you indicated you are using the EMCCLK.  If so, change the "Div-1" to "Div-2" or something slower.  Or use the CCLK (set EXTMASTERCCLK_EN to disable) and set the CONFIG_RATE to something slow like 36.4 as a test.

Clocking to End of Startup
By default, DONE is released in phase 4 of startup, and DONE_PIPE is enabled to add one
additional clock cycle of latency. DONE indicates that configuration is complete and all
data has been loaded, but some extra clock cycles need to be applied to ensure the startup
sequence completes correctly all the way to phase 7, End of Startup. A conservative
number for the clock cycles required after DONE is 24; this will account for the most
common use cases. The bitstream options LCK_cycle or Match_cycle will add an
undefined additional number of clock cycles.

== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
ddn
Moderator
Moderator
631 Views
Registered: ‎06-06-2018

@seamusbleu ,

From your configuration status register values, i feel everything is okay. Since you are connecting JTAG Cable, EOS Pin is high. If you dont connect JTAG Cable what's the status of EOS Pin ?

>> Use a simple design of Blinking LEDS/ counter and generate a BIN File for SPIx1 mode and use less CCLK Frequency and boot from flash and then share your configuration status register values.

>> Repeat the above step with your current design for SPIx1 and less CCLK Frequency and share the configuration status register values.

>> 

#1 Share the VCCINT,VCCBRAM,VCCAUX &VCCO(VCCO Banks of flash related pins) Scope shots (one below the another) when you didn't  connect JTAG Cable and when you connect JTAG Cable.

  Note : Do the above step when you boot from flash.

#2  If the clock driving BRAM coming from external source on clock capable IO, then share the Scopeshot of that clock when you didn't connect JTAG Cable and when you connect JTAG Cable.

   Note : Do the above step when you boot from flash.

#3 Share your Power distribution network for our reference.

     >> Needed to understand the JTAG Dependency on your power network.

#4 Share the configuration interface schematic of Flash and JTAG

 

Above information is needed to understand the issue.

Regards,
Deepak D N
---------------------------------------------------------------------------
Please Kudo and Accept as a Solution, If it helps.
---------------------------------------------------------------------------
0 Kudos
seamusbleu
Explorer
Explorer
620 Views
Registered: ‎08-12-2008

UG470 states  that persist option is not recommended for standard master SPI configuration modes (your mode).  I deleted your zip file, and I don't see it available anymore - but I think you had persist option set?  Another shotgun test would be to remove that (if I'm correct that it was there).  Ideally, you find some change that makes this issue go away so you can isolate the cause.  Good luck.

In response to @ddn 's post, the only way to monitor the EOS without a JTAG cable would be to instantiate the STARTUPE2 module to get access to the EOS pin, and then bring it outside your FPGA.  What would be really neat, but a fair amount of work, would be to read the status register through an ICAPE2 primitive and then send that value outside your FPGA.

== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
0 Kudos
jim_godfrey1
Visitor
Visitor
258 Views
Registered: ‎05-25-2021

Turns out I was sequencing power incorrectly for this testing.