UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Observer dylan.campbell
Observer
580 Views
Registered: ‎08-31-2018

Flash programming failed - Worked previously

Hello all,

I have somewhat of a weird issue I'm running into.  I have a board with an Artix-7 on it and the flash S25FL064LABMFI013.  This board has been happily working for weeks with no issues being programmed.  Then suddenly I started getting errors when I try to program the flash.  The first error I got was "[Labtools 27-3347] Flash Programming Unsuccessful: cannot set write enable bit or blocks(s) protected"

The strange thing about this I have the ability to write to the flash via the FPGA by our HDL already loaded.  So I know the flash can be written to(I've verified it is actually written to correctly).  What is also strange is that only some of the boards are experiencing this problem.  But some boards that were not experiencing this problem later developed it.  

I have tried programming the boards with different programmers, different computers.  I've tried different boards and quite a few have the problem.  But not all.  But again some of them developed the problem after a while.  I’ve tried different JTAG programming frequencies and different configuration frequencies.  I’ve taken the working programming from a good board and given it to a bad board and that doesn’t work.  Giving the good programming back to the good board still works. 

What’s more confusing is that I get different errors depending on the scenario.  Sometimes I just get a generic “Flash Programming Unsuccessful” message with no info. 

Programming the FPGA itself always works(IE bit file).

I know that was a lot of info that isn’t exactly well constructed but I have trouble connecting all of it into a coherent problem. 

Is there a way to get more info about the failure from Vivado?  I’m going to try replacing the flash part but I have a couple of days until they arrive and like to make a little progress in the meantime.

Thanks for reading!

0 Kudos
10 Replies
560 Views
Registered: ‎01-22-2015

Re: Flash programming failed - Worked previously

Hi Dylan,

Some random thoughts and things to check…

1) I see from Table D-1 in UG908 that the S25FL064L is approved by Xilinx for use with Artix-7.  So, good here.

2) I assume you are using flash in master SPI mode, M[2:0]=001?  If you are using SPIx4 then try instead to use SPIx1.

3) According to table 47 of S25FL064L datasheet, VCC(min) is 2.7V.  Measure and verify this on your board.

4) In Fig 2-12 of UG470, it is normal to have VREF=VCCO_0=VCC.  Table 2 of DS181 says that VCCO_0 should be between (1.14 – 3.465 V).  However, VCC(min)=2.7V for flash can modify this to (2.7 – 3.465 V).  Are you good here?

5) In Fig 2-12 of UG470,  measure that VCCAUX=(1.71-1.89 V) per table 2 of DS181.

6) As you know, programming the flash is done via JTAG.  The programmer module sends the configuration file via JTAG to the FPGA and the FPGA sends the file via SPI to the flash.   Use oscilloscope to view all digital lines of the JTAG interface and of the SPI interface.  Signals on these lines should be clean and toggling between the levels of (0.3 x VCC) and (0.7 x VCC).

7) In Fig 2-12 of UG470,  you can see connections M[2:0] of the FPGA.  If possible, use oscilloscope to monitor these lines as you power-up the board.  They should change quickly and then remain steady at the correct values (001).

8) Use oscilloscope to check that all other FPGA voltages are coming up to the proper levels and in the proper order according to the Power-ON Sequence shown on page 8 of DS181.

Cheers,
Mark

Observer dylan.campbell
Observer
520 Views
Registered: ‎08-31-2018

Re: Flash programming failed - Worked previously

Hi Mark,

Thank you for the detailed reply!

I've gone through your list and I'll summarize it below.

2) Yes I am using SPIx1 with M[2:0] = 001

3) I've verified that the VCC pin of the flash is seeing 3.3v

4) VCCO_0 is tied to 3.3v so I don't believe this is a problem.

5) I measured VCCAUX and VCCINT and they are both within the acceptable range.

6) All the SPI signals look good.  The JTAG signals look good except MISO.  I'm not familiar with JTAG(other than using it to program) so I'm not sure what is acceptable for the signals.  I attached a picture of what I'm seeing.  Sometimes it appears that the MISO signal is being driven high z.  Which makes sense but it seems to do it a lot and sometimes in what looks like the middle of a packet.  But maybe this is normal.

7) I probe all of the M signals during power up and they all looked clean.  They were stable once they reached their pull up value.

8) I checked all the FPGA voltages and they seem to conform to the specification.  Except the rise time of VCCINT.  Our rise time is 0.09ms while the min spec is 0.2ms.  We also do not sequence the supplies when it turns off.  They all turn off simultaneously.  

Thanks again for the help!  I've been stuck on this for a few days now with very little progress.  Is there anything in Vivado that report in more detail the problem?  Most the time I get the error "[Labtools 27-3161] Flash Programming Unsuccessful" which is pretty generic.  

Thanks,

Dylan

Artix_7_JTAG_MISO.jpg
503 Views
Registered: ‎01-22-2015

Re: Flash programming failed - Worked previously

Dylan,

Wow!  We need you working at our place.  When can you start? :-)

Your report of test results is very clear.  –except for MISO. 

  1. I assume that MISO is also pin 8 of the JTAG connector and that you have it connected TDO of FPGA as shown in Fig 2-12 of UG470?
  2. Do you have only the one flash and the FPGA connected to the JTAG connector (ie. no other JTAG devices on the board)?
  3. Can you send us *everything from the Labtools window where you are seeing the error message? (*everything =  from start to finish of your flash programming session)
  4. Has anyone been fiddling with eFUSEs for the FPGA?
  5. If you are generating the bitstream in Vivado, then open the implemented design and go to "Settings > Bitstream > Configure additional bitstream settings > Configuration > Configuration Pin Settings During User Mode" - and tell me the setting you see for "Unused IOB Pins".
  6. In the setup for Labtools, you will find a selection for "State of non-config mem I/O pins:".  Set this to "Pull-none" as shown below.

pull_none.jpg

Observer dylan.campbell
Observer
488 Views
Registered: ‎08-31-2018

Re: Flash programming failed - Worked previously

Hi Mark,

Hahah Thanks.  I’m trying my best!  I’m nearly at the point where I have no idea where to go next so I’m trying to thoroughly try everything I can. 

1) Sorry for the confusion. I was reading the label off our programmer but I read the SPI side instead of JTAG.  When I said MISO I should have said TDO.  As you said it is connected to pin 8 of the JTAG header.

2) That is correct. There is only one flash connected to the FPGA and only the FPGA on the JTAG.  There is a header for the flash but it is for debug purposes. 

3) I’m attaching my screenshots of the process. It starts with pressing ‘auto-connect’(not shown). Then I right click the FPGA in the hardware window and press ‘Add configuration memory device’.  Then when the memory device dialog comes up I select mine and press ok.  Then I right click on the flash part in the hardware window and select ‘program configuration memory device’.  Then I select my .bin file(I’ve tried mcs with the same results) and press ok.  Then it tries to program the FPGA and succeeds.  Then it tries to erase the flash and fails after a few seconds.

4) I’m very confident they haven’t been touched. Unless a rouge tech decided to mess with it.

5) Unused IOB is set to PULLDOWN

6) That setting is set and the problem persists.

Let me know if I missed something or something in the screenshots are unclear.

Thanks!

Dylan

1_after_auto_connect.PNG
2_after_select_add_configuration_memory.PNG
3_after_pressing_ok.PNG
4_after_pressing_program_memory_device.PNG
5_during_programming.PNG
6_on_failure.PNG
0 Kudos
474 Views
Registered: ‎01-22-2015

Re: Flash programming failed - Worked previously

Thanks for all the clear answers and photos!

More thoughts and observations…

1) Order another type of flash.  We’ve had very good luck with MT25QL128ABA1ESE-0SIT, which I believe will fit your board and is in-stock at Digikey.  This flash has 128Mb (whereas yours has 64Mb) but it should work for you.

2) With Labtools, I see you have tried to send both bin-file and mcs-file.  Using mcs-file is more conventional.  Let’s stick with using mcs-file for all future testing.

3) Error message in your screenshot that says “…cannot set write enable bit or block(s) protected”  This kinda sounds like some lock/security bits are getting inadvertently set in the flash.  I see from S25FL064L datasheet that such things are possible.  Since I doubt we can get more information from Labtools about this  -  let’s assume that corruption of signals being passed from FPGA to flash during programming of flash is causing this. 

4) Going (for now) with assumption that signal corruption is inadvertently locking the flash, let’s try the following:

  • find a board that is still working (or preferably one that has not yet been used)

  • unsolder-from-the-board the debug header for the flash, since this might be causing data corruption

  • put the following constraints in your Vivado project’s xdc constraints file (note that last constraint sets programming frequency to 3MHz)
    set_property CFGBVS VCCO [current_design]
    set_property CONFIG_VOLTAGE 3.3 [current_design]
    
    #CONFIG.SPI_BUSWIDTH: indicates PROM uses standard SPI communications (1bit-DIN, 1bit-DOUT)
    set_property BITSTREAM.CONFIG.SPI_BUSWIDTH 1 [current_design]
    
    #CONFIG.CONFIGRATE: specifies 3MHz clocking of the SPI interface with the PROM
    set_property BITSTREAM.CONFIG.CONFIGRATE 3 [current_design]
  • since you have HDL that can write bin-files to flash, it sounds like you may be doing multiboot.  Ensure that all your Multiboot constraints in the xdc file are commented out.
  • generate the bitstream, copy it to directory, C:/temp/, on your computer and rename it to my_bit.bit

 

  • convert my_bit.bit to an mcs-file called my_mcs.mcs by typing the following command into the Vivado Tcl console
    #NOTE: "-size" below is flash size in MBytes (so 64Mb flash is 8MB)
    write_cfgmem -format mcs -size 8 -interface SPIx1 -loadbit {up 0x00000000 "C:/temp/my_bit.bit" } -force -file "C:/temp/my_mcs.mcs"
  • Use Labtools/Vivado to send my_mcs.mcs to the flash

  • try sending my_mcs.mcs to flash many times – does  it always work?

5) Faithfully record all the error messages that LabTools is throwing at you - and record when they occur (eg. during flash erase, or program, or verify)

m

Observer dylan.campbell
Observer
447 Views
Registered: ‎08-31-2018

Re: Flash programming failed - Worked previously

1) Ok, I’ll look into getting some purchased.  I’ve also purchased some spares of the old one to try switching it out.  Maybe the problem follows the board and not the flash.

2) I’ll use MCS from now on. 

3) The write bit being inadvertently set is what it sounds like but I seem to be able to write to the flash perfectly fine.  Our own flash programming HDL even checks the values it wrote and has not reported any errors when writing to the flash.  I’m going to see if our software guys can give me a more detailed test.

4) I put in all those property to our build script and generated the MCS file like described.  I also took out all the multiboot stuff.  I’ve taken this and setup a tcl script to program a brand new(working) board few hundred times to see it ever fails.   As of now it’s successfully programmed a few dozen times.  It could be a while before it fails because one of the boards that started having this problem had been programmed at least 100 times before it exhibited this problem. 

There's an option when programming to only erase.  So I thought maybe we can narrow it down by seeing if it can be erased  So I tried only erasing the flash without even providing a .mcs file.  It still fails here.  So it suspect it doesn’t have much to the contents of the .mcs file.  I may have forgotten to mention but throughout this whole problem the error always occurs during the erase phase.  It happens pretty early into the erase process(1-2 seconds).

I have a different JTAG programmer that I'm going to try also.

Thanks again for the continued help!  

0 Kudos
Moderator
Moderator
417 Views
Registered: ‎06-05-2013

Re: Flash programming failed - Worked previously

Thanks @ markg@prosensing.com for helping here. Lot of great feedback and debugging tips.

@dylan.campbell
Thanks for adding details. I was trying to emulate the same using SVF. So your flash MFG ID, capacity and Memory type seems to be correct. Can you try the following just to see what errors comes out when SVF is executed?

> Open HW manager and Go to Tools> Create SVF target>
> Add your A7 device and flash memory.
> Add the flash programming operations.
> Export SVF file and close the SVF target.
> Now connect to your JTAG chain with flash added and execute the svf file using the below command:
>>> execute_hw_svf -verbose C:/yourfilename.svf

Try to add different flash operations(erase,prorgam, verify, blank check) by creating different SVF file. And confirm where it gets stuck. -verbose option might give you some specific error.

Another test which you can try if certain location are protected. (Assuming if that's the case) Try to generate a mcs file with some upper address location like 0x04000000 and see if that's get program.

Also you can follow the AR https://www.xilinx.com/support/answers/61067.html script and try to see outcome of Steps 5 & 6. This might confirm the configuration settings.

Thanks
Harshit
-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
Tags (1)
Highlighted
Observer dylan.campbell
Observer
401 Views
Registered: ‎08-31-2018

Re: Flash programming failed - Worked previously

Hi Harshit,

I tried what you suggested with the SVF files.  I made 3 SVF files.  One with erase, program and verify enabled.  One with only erase and one with only program.  All three of these fail.  Below is the output of one.  They all fail in the same way.

execute_hw_svf -verbose C:/Users/CampbellDy/Desktop/configs/flash_test.svf
INFO: [Labtoolstcl 44-615] Merging header information for SVF file. This may take a while depending on your SVF file size.
INFO: [Labtoolstcl 44-548] Creating JTAG TCL script from SVF file. This may take a while depending on your SVF file size.
INFO: [Labtoolstcl 44-549] Re-opening target in JTAG mode
INFO: [Labtools 27-3283] Add new devices to SVF target using the create_hw_device command
set current_hw_jtag [get_property hw_jtag [get_hw_target localhost:3121/xilinx_tcf/Xilinx/svftarget_0]]
INFO: [Labtoolstcl 44-551] Sourcing JTAG TCL script: C:/Users/CampbellDy/AppData/Roaming/Xilinx/vivado_lab/.Xil/vivado_lab-13808-WIL07215L/flash_test.tcl
source C:/Users/CampbellDy/AppData/Roaming/Xilinx/vivado_lab/.Xil/vivado_lab-13808-WIL07215L/flash_test.tcl
# set_property ENDSTATE_IR {IDLE} $current_hw_jtag
# set_property ENDSTATE_DR {IDLE} $current_hw_jtag
# run_state_hw_jtag RESET
# run_state_hw_jtag IDLE
# scan_ir_hw_jtag 6 -tdi 09
# scan_dr_hw_jtag 32 -tdi 00000000 -tdo 0362E093 -mask 0FFFFFFF
ERROR: [Labtools 27-2255] Output value 00000000 does not match the tdo option value
WARNING: [Labtoolstcl 44-546] Execute SVF failed during JTAG TCL script sourcing
INFO: [Labtoolstcl 44-550] Restoring target to original mode
INFO: [Labtools 27-3283] Add new devices to SVF target using the create_hw_device command
execute_hw_svf: Time (s): cpu = 00:00:03 ; elapsed = 00:00:51 . Memory (MB): peak = 1854.523 ; gain = 82.281

I'm not really familar enough with SVF files and their uses to make much of this.

Next I tried using a .mcs file with the bitstream starting at 0x400000 and I got the same error as before(failed due to write protect bit).  

I ran the program in that AR and below are the results before trying to program the flash.

xspi_read_id
0 = A956
1 = A956
2 = 0050
3 = FF01
4 = 6017
5 = FFFF
6 = FFFF
7 = FFFF
8 = 0000
9 = 0000
10 = 0000
11 = 0000
12 = 0000
13 = 0000
14 = 0000
15 = 000E
Manufacturer ID : 01
Device ID
- memory type : 60
- memory capacity: 17
xspi_read_statregs
Status Register 1:
- bits [7:0] : 04
Status Register 2:
- bits [7:0] : 00
Configuration Register :
- bits [7:0] : 00

And after trying to program the flash:

xspi_read_id
0 = A956
1 = A956
2 = 0050
3 = FF01
4 = 6017
5 = FFFF
6 = FFFF
7 = FFFF
8 = 0000
9 = 0000
10 = 0000
11 = 0000
12 = 0000
13 = 0000
14 = 0000
15 = 000E
Manufacturer ID : 01
Device ID
- memory type : 60
- memory capacity: 17
xspi_read_statregs
Status Register 1:
- bits [7:0] : 06
Status Register 2:
- bits [7:0] : 00
Configuration Register :
- bits [7:0] : 00

I'm not sure what those status registers mean or if they are correct or not.  Please let me know if there is something you'd like me to do to get more information.  

Hopefully something there help.

Thank you!

Dylan

 

 

Observer dylan.campbell
Observer
392 Views
Registered: ‎08-31-2018

Re: Flash programming failed - Worked previously

I've run a couple more tests in the meantime.  

I tried a completely different programmer and the problem persists in the same exact way.

I replaced the flash chip with a brand new one and now I can program it.  

So it does seem to follow the flash part.  I still need to talk to the software guys and ask about how they program the flash.

Thank you,

Dylan

Moderator
Moderator
380 Views
Registered: ‎06-05-2013

Re: Flash programming failed - Worked previously

Thanks Dylan for running those test. And testing the new flash. From the registers it seems block protection is enabled on those flashes. Here is the status register1 breakdown:-

In your case Bit[2] is programmed(value x06). And as per the status register table, this suggest BP0 is programmed which is a block protection bit. Here is the snapshot from flash datasheet:-

Flash_status_reg1.JPG

flash_block protection.JPG

 

Yes it would be helpful to check with software guys what else they are programming. 

Thanks

Harshit 

 
-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos