UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Observer risamu
Observer
2,727 Views
Registered: ‎01-08-2018

Vivado 2015.2 Producing Inconsistent Checksum

I am unable to achieve consistent results with some builds on Vivado 2015.2 (Win7 64bit, targeting Zynq-7020). Specifically, when I attempt to repeat a build from the same Block Diagram TCL source, using the same build scripts, on the same machine. I am getting different checksums of the resulting MCS file, as reported by the write_cfgmem  TCL command.

 

On some larger designs I am seeing 1 "most-frequent" checksum on roughly 50% of builds, and then up to 8 other checksums less frequently (out of 155 builds). I've noticed that the auto-generated AXI Interconnect (2.1) will generate inconsistently when it contains 8 master AXI ports (difference is visible even before Synthesis, in the Block Diagram), but even when NO interconnect has more than 8 master ports, the checksum issue persists. I believe I've narrowed the issue down to a simpler build with only a Zynq 7000 PS and two AXI 1G/2.5G Ethernet Subsystems (7.0) with corresponding DMAs. That small build reports ~80% consistency on the checksum. I am able to get a consistent checksum with similar designs that do not include the AXI 1G/2.5G Ethernet Subsystem.

 

The divergence between builds is apparent as early as Phase 1 of implementation at which point the logged checksums are already inconsistent. I've also compared the post-synthesis netlists and confirmed that they are not identical.  This leads me to believe this is not merely an implementation/place-and-route issue.

 

Per AR#61599, Xilinx claims "Vivado should generate identical results between runs involving identical: Design sources, Constraints, Tcl scripts and command sequences, Tool and command options, Vivado software version, Multi-threading settings, Operating Systems."  I've kept all of the above consistent, and yet still am having this issue. That AR also requests that a Service Request be opened with a test case to demonstrate the problem; however, I am redirected to this forum whenever I attempt to submit a Service Request with my test case about this issue. 

 

 

Has anyone else seen this issue? Is there something wrong with the AXI Ethernet IP? Any workaround to produce a consistent checksum?

 

Please advise.

0 Kudos
29 Replies
Scholar hbucher
Scholar
2,715 Views
Registered: ‎03-22-2016

Re: Vivado 2015.2 Producing Inconsistent Checksum

@risamu As far as I know, implementation is an optimization method that could indeed produce different results depending on several factors. I don't think it is supposed to give you the same results every time.

Check out this AR:

https://www.xilinx.com/support/answers/61599.html

 

vitorian.com --- We do this for fun. Always give kudos. Accept as solution if your question was answered.
I will not answer to personal messages - use the forums instead.
0 Kudos
Observer risamu
Observer
2,710 Views
Registered: ‎01-08-2018

Re: Vivado 2015.2 Producing Inconsistent Checksum

@hbucher Thanks for the response. This might have been unclear earlier: the problem I'm describing appears before implementation.

 

 I've also compared the post-synthesis netlists and confirmed that they are not identical.  This leads me to believe this is not an implementation/place-and-route/optimization issue.

 

 

Per AR#61599, this should be consistent. I'm running this on the same machine, using the same sources/scripts:

* Same device sources

* Same Constraints

* Same TCL scripts and command sequences

* Same Vivado SW Version

* Same Multi-threading Settings

* Same Operating System

 

 

0 Kudos
Scholar hbucher
Scholar
2,709 Views
Registered: ‎03-22-2016

Re: Vivado 2015.2 Producing Inconsistent Checksum

@risamu Where exactly it guarantees repeatability in that document? 

vitorian.com --- We do this for fun. Always give kudos. Accept as solution if your question was answered.
I will not answer to personal messages - use the forums instead.
0 Kudos
Observer risamu
Observer
2,703 Views
Registered: ‎01-08-2018

Re: Vivado 2015.2 Producing Inconsistent Checksum

From the first few lines of AR#61599 (emphasis mine):

 

"Description:

Are Vivado results repeatable for identical tool inputs?

 

Solution:

For the most part the answer is yes, Vivado should generate identical results between runs involving identical [things I've listed as being identical in my case]"

0 Kudos
Scholar jmcclusk
Scholar
2,688 Views
Registered: ‎02-24-2014

Re: Vivado 2015.2 Producing Inconsistent Checksum

I have also observed synthesis netlists that vary from run to run.   Synthesis has been multi-threaded for awhile now, and this is the root cause.    I'll bet if you set a single thread:

 

set_param general.maxThreads 1

Then you'll get an identical netlist on successive runs. 

Don't forget to close a thread when possible by accepting a post as a solution.
0 Kudos
Scholar hbucher
Scholar
2,686 Views
Registered: ‎03-22-2016

Re: Vivado 2015.2 Producing Inconsistent Checksum

@risamu "For the most part..." - that is the opposite of a guarantee.

vitorian.com --- We do this for fun. Always give kudos. Accept as solution if your question was answered.
I will not answer to personal messages - use the forums instead.
0 Kudos
Observer risamu
Observer
2,660 Views
Registered: ‎01-08-2018

Re: Vivado 2015.2 Producing Inconsistent Checksum

@jmcclusk I've tried that. The issue persists even with that command in my project build TCL. 

 

I even verified the setting was correctly applied by using get_param in the built project. I'm running repeated builds with the parameter set now, but so far I'm getting the same number of inconsistent checksums as before. 

0 Kudos
Scholar hbucher
Scholar
2,650 Views
Registered: ‎03-22-2016

Re: Vivado 2015.2 Producing Inconsistent Checksum

 @risamu 

From the docs I conclude that you cannot guarantee 100% repeatability, you cannot rely on that for whatever reason you need.

Can I ask you why you need 100% repeatability?

The issue is that the build will vary if the number of cycles involved in a given step changes from build to build. 

This also is affected by memory cache but Windows has too much more clutter preempting your process than Linux.

Have you tried the build on Linux? You might get better results there.

vitorian.com --- We do this for fun. Always give kudos. Accept as solution if your question was answered.
I will not answer to personal messages - use the forums instead.
0 Kudos
2,645 Views
Registered: ‎01-08-2012

Re: Vivado 2015.2 Producing Inconsistent Checksum


@hbucher wrote:

Can I ask you why you need 100% repeatability?


I can't speak for the OP, but some industries, possibly those involving safety, money, military or spies, may require source code audits as well as some proof that the binary image really was derived from the source that was audited.

 

That's difficult to achieve without repeatable builds.

0 Kudos
Observer risamu
Observer
2,528 Views
Registered: ‎01-08-2018

Re: Vivado 2015.2 Producing Inconsistent Checksum

@hbucher from the same docs I conclude that Vivado should produce consistent results. Apparently, we disagree on how to interpret that AR... maybe someone from Xilinx can clarify?  I am able to get 100% repeatability on builds some builds--it seems reasonable to me to expect the same repeatability on others.

 

On the matter of why, @allanherriman is correct. We utilize a checksum to provide traceability of the bitstream back to the sources that generated it. 

0 Kudos
Scholar hbucher
Scholar
2,522 Views
Registered: ‎03-22-2016

Re: Vivado 2015.2 Producing Inconsistent Checksum

@risamu Perhaps it is an English language problem but "most of the time" in no way means "100% of the time" as far as I can tell, especially when your own builds are telling you exactly that.

But yes, wait for the feedback from an official source. Perhaps you can put a service request for that.

https://www.xilinx.com/support/answers/64454.html

 

vitorian.com --- We do this for fun. Always give kudos. Accept as solution if your question was answered.
I will not answer to personal messages - use the forums instead.
0 Kudos
Scholar jmcclusk
Scholar
2,518 Views
Registered: ‎02-24-2014

Re: Vivado 2015.2 Producing Inconsistent Checksum

Hi @risamu,  Did you try setting the single threading parameter in GUI project mode?   I'm not at all sure that setting this parameter in the parent process will be carried over into the synthesis sub-process in GUI mode.  I would only trust the single thread setting when running Vivado in non-project mode (batch mode).    

Don't forget to close a thread when possible by accepting a post as a solution.
0 Kudos
Highlighted
Moderator
Moderator
2,501 Views
Registered: ‎07-21-2014

Re: Vivado 2015.2 Producing Inconsistent Checksum

@risamu

 

Can you please let us know below details to debug this issue further:

1. Which OS are you targeting?

2. Please share the log file which shows different checksum value for the same design. Also, attach the Vivado synthesis log files here.

3. As you have mentioned in your initial post that you are suspecting some issue with the AXI Ethernet IP, can you please elaborate on this?

4. Can you please share a test case? I will share a private link to upload the design files.

5. Did you apply any patches for the current Vivado version or for the project?

 

If the design is same(constraints, RTL, IP configuration and project settings) then Vivado should deliver same results for every run.

 

Thanks,

Anusheel

 

Thanks
Anusheel
0 Kudos
Moderator
Moderator
2,493 Views
Registered: ‎07-21-2014

Re: Vivado 2015.2 Producing Inconsistent Checksum

@risamu

 

Did you try using any of the latest Vivado version for your project? If possible, can you please try in 2017.4/2017.3 and let us know the results.

 

Thanks,

Anusheel

Thanks
Anusheel
0 Kudos
Observer risamu
Observer
2,440 Views
Registered: ‎01-08-2018

Re: Vivado 2015.2 Producing Inconsistent Checksum

Hi @anusheel,

 

1. I am targeting Windows 7, 64-bit.

2. I have attached the synthesis log files for 4 checksum-divergent runs to this post. Run_1's checksum occurred 88% of the time, Run_2 got 10%, Run_50 appeared only twice (0.72%), and Run_296 produced a unique checksum that only appeared once in 276 runs (0.36%).

3. My test build includes only a Zynq PS block and 2 Ethernet Subsystems. Without the Ethernet subsystems, I can get a consistent checksum--even on much more complex designs.

4. I have a test case, more log files, and build scripts ready to send to you. I attempted to submit a Service Request (as instructed in AR#61599) but was redirected to this forum instead. Please send me the link and I will upload all the files.

5. I have not applied any patches to 2015.2. I am aware of "Vivado 2015.2 Update 1" but, on the Vivado downloads page, Xilinx recommends to continue using 2015.2 for my device (Zynq 7020) rather than install the update.

 

I have not tried Vivado 2017.3 nor 2017.4. Unfortunately, I do not have time to try those versions right now. 

0 Kudos
Moderator
Moderator
2,428 Views
Registered: ‎07-21-2014

Re: Vivado 2015.2 Producing Inconsistent Checksum

@risamu

 

I can see that there is a difference in "Report Cell Usage" where LUT count is different between the runs which might be resulting in different checksum for each implementation runs.

 

There are 8 critical warnings and 419 Warnings, did you get a chnace to look into it? As the shared log does not have the details of these warnings, I was not able to check at my end.

 

 

Also, I can see that you are not using default value for maxThreads, can you please let me know what was the value when you ran all these runs?

INFO: [Common 17-600] The following parameters have non-default value.
general.maxThreads

 

I have sent you an Ezmove to upload the project.

 

Thanks,

Anusheel

Thanks
Anusheel
0 Kudos
Observer risamu
Observer
2,418 Views
Registered: ‎01-08-2018

Re: Vivado 2015.2 Producing Inconsistent Checksum

 

@anusheel I have submitted the project through the Ezmove link you sent me.

 

The non-default value for maxThreads is due to my project TCL which executes:

 

set_param general.maxThreads 1

  

 This was intended to force single-threaded execution, hoping that would solve the synthesis inconsistency.

 

The critical warnings are relating to the auto-generated constraints for the Ethernet IP which no longer apply because the ports are not top-level pins at the Ethernet IP: I am utilizing an MII/RMII converter block to reduce the number of pins used on the package. I have re-created the constraints on the appropriate top-level pins, but I don't know how to prevent the IP from trying to auto-generate its own constraints.

 

 

0 Kudos
Moderator
Moderator
2,363 Views
Registered: ‎07-21-2014

Re: Vivado 2015.2 Producing Inconsistent Checksum

@risamu

 

I ran around 10 runs on my Linux machine for the test case "cksm_tst_small.xpr.zip" and then compared the synthesis results. All the run log including the utilization is same for all, I guess either the script you are sourcing is resulting into this issue or this is a machine specific issue.

 

Did you try using any Linux machine or any other machine to check if this is the case? Also, what happens when you only use Vivado "no build scripts" to launch the runs?

 

Thanks,

Anusheel 

Thanks
Anusheel
0 Kudos
Observer risamu
Observer
2,353 Views
Registered: ‎01-08-2018

Re: Vivado 2015.2 Producing Inconsistent Checksum

@anusheel It's possible that part (or the whole) of the issue may be occuring during the generation of the block diagram from TCL script.. The BD Tcl script I use was auto-generated using the "write_bd_tcl" command. I am not able to reproduce the issue on successive manual-Vivado- "no build scripts" launches on this small test project.  But I have not run more than 7 or 8 successive runs manually. 

 

 

I do not have access to a Linux machine to test. Can you execute the "Build_clean_repeat.py" script I uploaded on a Windows 7 machine? It calls all the necessary project scripts and should allow you to reproduce this issue I'm having.

 

If you cannot run the python script, can you please perform the runs by using the following command:

 

 

vivado -source release_from_tag_build.tcl

 

 

A few of those runs should demonstrate the issue.

 

 

0 Kudos
Moderator
Moderator
2,311 Views
Registered: ‎07-21-2014

Re: Vivado 2015.2 Producing Inconsistent Checksum

@risamu

 

I used the shared python script and also the Tcl script to launch runs in order to reproduce the issue on my Windows7 64-bit machine with Vivado 2015.2, but I still don't see any differences in the synthesis runs at my end.

 

Please share below details to understand the issue:

1. Can you check with any latest Vivado version?

2. Did you get chance to try on different machine?

3. Are you writing new block design tcl every time or sourcing the same file for multiple runs?

4. Is this issue related to number of runs(any minimum number?) or the same issue is reproducible with 2-3 runs at your end?

 

Thanks,

Anusheel

Thanks
Anusheel
0 Kudos
Observer risamu
Observer
2,300 Views
Registered: ‎01-08-2018

Re: Vivado 2015.2 Producing Inconsistent Checksum

@anusheel 

 

The python script is just a tool for automation to demonstrate the problem. I don't normally use that python script except to investigate whether builds are consistent or not.

 

1. I have not tried different Vivado Versions, most of our licenses extend only to 2015.2. 

 

2. I have successfully reproduced the problem on a different, albeit still Windows 7 64bit, machine.

 

3. I am sourcing the same block design TCL (design_1.tcl) every build. 

 

4. This is a very intermittent and unpredictable problem: On a more complex design, I've seen the checksum be different almost every other run. On the design I shared with you, however, I see around 85% consistency on the checksum, so it could take 10 to 20 runs to see it using the python script I shared. This design is the simplest design I could share that still produces the problem. Please look at the attached log (which I shared with you in the BCR_log_11Jan18_1653 folder) to get an idea of the frequency of the problem in this project.

0 Kudos
Scholar hbucher
Scholar
2,296 Views
Registered: ‎03-22-2016

Re: Vivado 2015.2 Producing Inconsistent Checksum

@anusheel I have been following this thread for a while - trying to help out.

According to my reading, this thread is moot because there is no guarantee that the checksums will be the same. 

 

 

vitorian.com --- We do this for fun. Always give kudos. Accept as solution if your question was answered.
I will not answer to personal messages - use the forums instead.
0 Kudos
Moderator
Moderator
2,280 Views
Registered: ‎07-21-2014

Re: Vivado 2015.2 Producing Inconsistent Checksum

@hbucher

 

Thanks for your inputs.

 

If the design is same and all the runs are launched without any changes in command/switches/settings then Vivado must deliver exact same netlist. Here the utilization report clearly shows the difference in utilization of LUTs which is an incorrect behavior.

 

We have seen few machine specific issues, as I am not able to reproduce yet I am not sure whether this is machine specific or a corner case issue with Vivado 2015.2.

 

I have received more inputs from @risamu on how to reproduce the issue and soon I will post a summary of this issue here.

 

Thanks,

Anusheel

Thanks
Anusheel
0 Kudos
2,175 Views
Registered: ‎01-08-2012

Re: Vivado 2015.2 Producing Inconsistent Checksum

Just by coincidence, a few weeks ago I had a customer come up with a requirement that they be able to build our source code and produce an identical binary image.  I've been preparing scripts to allow them to do this, and my results have been mixed.

 

I have found no build-to-build variations on a single machine, i.e. all builds on the same machine produced the same binary.

I have access to three test machines, and found build-to-build variations between machines.

 

Vivado version: 2015.4.

 

Machine 1.  i7-3970X, Windows Server 2008 R2 (64 bit)

Machine 2.  i7-5930K, Windows Server 2012 R2 (64 bit)

Machine 3.  i5-6600, Ubuntu 16.04.3 (64 bit)

 

All other conditions are identical, and the scripts contain

set_param general.maxThreads 1

 

Yes, I know that these operating systems aren't on the recommended OS list for 2015.4.

 

Machines 1 and 3 always give the same results.  Machine 2 always gives different results to Machines 1 and 3.  The differences first show up in synthesis, and then diverge from there.

 

 

My problem is that I don't control the customer's machine, and I can't see a way of making binary images that are guaranteed to match the ones generated by the customer.

How should I proceed?

 

EDIT: would it be sufficient to disallow the use of Windows 8 and later (which would include Server 2012) ?  It's clearly not a Windows vs Linux thing, as I had identical results between Windows Server 2008 (= Vista) and Ubuntu.  Is there some DLL different between the two versions of Windows that I tried that would influence the result?

 

Thanks,

Allan

Scholar hbucher
Scholar
2,171 Views
Registered: ‎03-22-2016

Re: Vivado 2015.2 Producing Inconsistent Checksum

@allanherriman I'd be interested in seeing, for each machine:

 

1. RAM available for Vivado

2. type of disk (benchmark?) where vivado and project are stored

3. execution time for synthesis and elaboration - for the same project

 

vitorian.com --- We do this for fun. Always give kudos. Accept as solution if your question was answered.
I will not answer to personal messages - use the forums instead.
0 Kudos
2,164 Views
Registered: ‎01-08-2012

Re: Vivado 2015.2 Producing Inconsistent Checksum

Do you think the timing makes a difference, as if the synthesiser looks at the elapsed time and decides to skip some steps if it thinks it's taking too long?

 

 

1.  RAM: enough.  Machine{1,2,3}:  {32GB, 96GB, 16GB}.  It's a small FPGA and Vivado didn't come close to using even half the available RAM even in the 16G case.

 

2.  I don't have benchmarks for the disks, but all are SSDs.  The Windows boxes (Machines 1 and 2) have separate SSDs for C: (where OS & Vivado are stored) and D: (where the FPGA files are held).  The Linux box (Machine 3) uses a single SSD for everything.

 

3.  Not sure about execution times.  I'll have to check the logs to get exact times, but Machine 3 was by far the fastest, and Machine 2 (the one the made the non-matching images) was marginally slower than Machine 1.

 

 

Perhaps my next step should be to get the IT guy to rustle up some more old test machines.

0 Kudos
Scholar hbucher
Scholar
2,162 Views
Registered: ‎03-22-2016

Re: Vivado 2015.2 Producing Inconsistent Checksum

@allanherriman Not synthesis, no. But elaboration is certainly an optimization problem. You can see the build strategies define "effort" for each one. But I have no visibility other than what it is described in the manuals - and that is inconclusive in both ways.

vitorian.com --- We do this for fun. Always give kudos. Accept as solution if your question was answered.
I will not answer to personal messages - use the forums instead.
0 Kudos
2,070 Views
Registered: ‎01-08-2012

Re: Vivado 2015.2 Producing Inconsistent Checksum


@allanherriman wrote:

Perhaps my next step should be to get the IT guy to rustle up some more old test machines.


The IT guy was really scraping the bottom of the barrel when he came up with:

 

Machine 4: i5-750, 16GB RAM, Ubuntu 16.04.3.

 

This (slowly) produced the same image hashes as Machines 1 and 3.

 

 

There we have it: the fastest and slowest machines gave the same results for my (single threaded) test case.

0 Kudos
2,039 Views
Registered: ‎01-08-2012

Re: Vivado 2015.2 Producing Inconsistent Checksum

I tested different operating systems on Machine 3.

 

Previously, when Machine 3 was running Ubuntu 16.04.3, it gave results identical to those of Machine 1 (which was running Windows Server 2008 R2) and Machine 4 (which was running Ubuntu 16.04.3).

 

Now, with Machine 3 running Windows 2012 R2, it gives results identical to those of Machine 2 (which was running Windows Server 2012 R2).

 

So it's the OS that makes the difference in my single-threaded Vivado 2015.4 example.

I assume that it's actually some DLL that's effecting the results, rather than the OS itself.  Would someone from Xilinx like to comment on that?

0 Kudos