Spartan3AN boot fail at cold

[sean.willis@lmco.com]

I'm working a production line of custom CCAs with Spartan3AN on them. They program and boot successfully at ambient but I'm having issues with several booting at cold (-40c). (From internal flash)

The power supply levels, ramp rates, and timing are the same at cold as at ambient.

Prog_b and Init_b go high about 60ms after power is applied, as expected, however Init_b goes low without the done pin going high thus signaling a failed boot.

When this occurs, before cycling power, the JTAG can scan the chain and successfully see the FPGA, however it cannot program/erase/readback flash.

A power cycle allows the FPGA to boot immediately, however after soaking a further 10-15 min the FPGA will fail to boot again.

I’m wondering if there is an issue with the internal flash locking up at cold.

We have tried lowering the config_rate setting, changing Vs pins from "Fast Read" to "Read", enabling "debug_bitstream", gutting the FPGA firmware to contain only a LED heartbeat, manually delaying Prog_b further after power up, and monitoring all voltages and clocks related to the FPGA & configuration. So far, no success.

We're now seeing a significant drop out rate and need to resolve this quickly.

Do you have any suggestions?

[u4223374]

sean.willis@lmco.com Just checking the two obvious points:

(1) Are you using industrial-grade chips from Xilinx's official supply chain? There hasn't been an accidental substitution of commercial grade chips, or "industrial" ones bought from unofficial suppliers? If either has happened, absolute #1 step is to buy ten from Avnet or Digikey and see if those work.

(2) If these are BGA chips, have you had one of the failed samples thoroughly inspected (eg. X-ray analysis of the balls) to verify that it's soldered properly?

[sean.willis@lmco.com]

Thanks for the reply. 

1) They are industrial-grade parts. All parts are marked with 5C/4I and are used as standard speed. All components have been procured through approved (reputable) vendors. 

2) These are BGAs and I have not run X-Rays at this point. These were assembled on the same assembly line as the rest of our CCAs using approved procedure. I tend to lean away from suspecting solder issues as there are several failing units and they all boot reliably at all temperatures except cold. Also if there were manufacturing issues, I  believe there'd be more reliable failures even at cold. 

[sean.willis@lmco.com]

I'm still looking for a solution. We're currently manually screening CCAs at cold. Attached are scope shots of the configuration at ambient and at cold for comparisons. Any other ideas?

[simon]

Hi sean.willis@lmco.com,

I am wondering what is the external pullup/down resistor value? e.g. mode pin, VS pin etc.

Thanks

Simon

[kkn]

sean.willis@lmco.com,

additionally can you capture the cclk after the power up on the fpga.

capture the status registers of the fpga for both after failure and passing scenario.

[sean.willis@lmco.com]

@simon, @kkn,

   I'm attaching an updated powerpoint with the requested data & CCLK screenshots. Also attached are the status register logs from impact. Note the status register reads only complete successfully if the FPGA has already booted. Otherwise the status reads freeze. Also JTAG access to flash freezes. The only recovery is to cycle CCA power. 

Also the CCA I used to capture the status registers has has VS modified to 101 for normal reads as a debugging step. VS of 111 failed originally on this card. 

The failing FPGAs are from this lot:

FG676AQ1617

D5229302A

5C/4I

[simon]

Hi sean.willis@lmco.com,

Do you isolate the power module@cold? Have you tried external power to see if the configuration is successful or not.

Thanks

Simon

[simon]

additionally, how to connect PUDC_B?

[sean.willis@lmco.com]

@simon,

    The power supplies are not isolated at cold. The VCCO & VCCAUX are driven by 3.3V liner regulator and VCCINT is driven by 1.2V linear regulator. I have not attempted to use external power supplies because I'd rather not back-feed the linear regulators. 

PUDC_B and SUSPEND are tied directly to ground. 

[kkn]

Hi Sean,

Since you are unable to read the status registers at -40C, can you try to scope the mode pins and see if they are stable at "011" and also VS pins if they are stable at "101" after powerup.

And also can you try to reduce the cclk to default frequency of 2Mhz if this is not tried to see if this helps in configuring.

--Krishna

[sean.willis@lmco.com]

Hi Krishna,

The mode pins and VS pins are tied directly to power or ground planes so they're definitely stable. 

I've reduced the cclk frequency to 2MHz by setting the "configrate" setting to 6. I've also tried a "configrate" of 3. Neither one resolved the problem. 

-Sean

[polyee13]

Sean from our team put out this post regarding the S3 3AN failure to boot at cold. I'm folllowing up on this issue because it is a really hot topic right now for our production team. We are in need of support to deterrmine root cause of this issue

The only thing I would add to the presentation that Sean created was our recent test data. Please see attached figures. One shows the init_b behavior during a pass condition, the other during a fail condition. The 3.3V measurement that you see is a probe anomaly. We confirmed 3.3V using a DMM is it's valid. 

As you can see the FPGA never configures. We brought up iMPACT and read the boot status register and it also indicated DONE not high, and that the SYNC word isn’t found (I confirmed the SYNC word exists in the bitstream). MODE pins are sampled correctly.

We attempted to manually initiate a configuration by momentarily grounding the PROG_B pin and still couldn’t get the device to configure. By this time temp was already HOT (70C).  

We then decided to modify the test station that the UUT stays on and we can do further actions in iMPACT.

The most interesting result is that I performed a Blank Check on the part, which came back as not blank and miraculously the FPGA configured immediately after that. On a second failure, I performed a get Device ID and it also configured immediately after issuing that command.

However, reading the boot status register does not initiate a configuration.

Keep in mind by this time the temperature was at HOT (appx 70C).

impact log fail

config failconfig passdrive prog_b manually