cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
migry
Observer
Observer
3,804 Views
Registered: ‎05-07-2020

New Ryzen 9 PC + New Vivado 2020.2 = crash, crash, crash :-(

Jump to solution

I've just upgraded my PC to a new AMD Ryzen 9 (Win10 Home, 64Gb). One thing I was looking forward to is shorter FPGA source compilation time.

On the old PC I think that I had an old version (2016?) installed. After some thought I decided to install the very latest version and during installation chose the "Webpack Edition" as I am a hobbiest/maker.

Once installed I went to a copy of the last project I was working on. It compiled on the old PC a few days ago. I was interested to see how much quicker compilation time would be. I tried "run synthesis" and the PC crashed and re-booted. The xpr file was corrupted, and had to be re-created. OK, so this time I deleted all the old "temporary" folders in case there was an issue with leaving the files from the older version of Vivado. Over the next 20 minutes I had at least 3 more PC crash and reboots, there was no consistency in where it got to. In one case it nearly got to generate a binary but I had forget to include the xdc. Once again the xpr was corrupted.

In the Windows event viewer logs I saw multiple entries for "Event ID 6008" before the entry for the reboot. The log entry was not helpful and did not point to Vivado, however I googled and found "AR#71064 2018.1 Vivado - Vivado crashes when opening a project or adding an HDL source file to a project" which mentioned Event ID 6005.

1) is this a known problem?

2) is there a fix?

3) is it related to AR#71064 for which a "patch" was provided. Since the above bug only mentioned 2017 and 2018, I wondered if the bug had been fixed in later versions.

I would try an earlier version, but the download and install is very very slow. I was getting 5Mb/s download.

Fingers crossed someone can help.

From the last crash there are files left in the "synth_1" sub-folder. I took a look at them, but didn't spot any crash related messages.

[Originally posted in the wrong sub-forum]

 
Tags (3)
0 Kudos
1 Solution

Accepted Solutions
mfreesto
Visitor
Visitor
1,466 Views
Registered: ‎05-31-2021

Hello all,

Another update... It turns out this SMT issue been fixed in the latest AMD AGESA version. This version was incorporated into the latest BETA BIOS from my motherboard vendor. After updating my BIOS to this BETA version I can now happily run Vivado with SMT enabled. It sees all 24 threads! Vivado still does not make use of them but that's another story. The moral of the story - keep one's BIOS up to date!

View solution in original post

43 Replies
Rmccarty
Explorer
Explorer
2,674 Views
Registered: ‎09-05-2020

Why are you using Windows on a new computer? It's just crap upon crap.

In my experience with Vivado 2020.1 and Ubuntu, it can be a bit unstable but so far only stopping and restarting Vivado fixes it.

richardhead
Scholar
Scholar
2,649 Views
Registered: ‎08-01-2012

@Rmccarty not a helpful reply.  We run windows and Linux machines and the compile times are comparable, with some people reconing windows builds are giving better results (anecdotally)

@migry we have no issues running on windows.  Do you have enough ram for the part you're compiling. Is it possible you have faulty ram module or motherboard?

watari
Professor
Professor
2,649 Views
Registered: ‎06-16-2013

Hi @migry 

 

It seems thermal issue on your PC.

Did you make sure your cooler system, power supply, temperature on CPU die and TPD value ?

 

Also, would you try to synthesis with small cpu number ?

 

Best regards,

migry
Observer
Observer
2,630 Views
Registered: ‎05-07-2020

Hello @richardhead (Hmmm there was someone on my university course with the same name, many many years ago!).

This is a brand new powerful PC (from a well known UK supplier) with fast CPU and plenty of RAM, 64Gb to answer your question.

Experimenting with FPGA projects is a hobby and perhaps the main thing for which I use a PC, which could be helped by a fast CPU. 

I have downloaded and run a number of benchmarks (including stress tests). If there was a CPU or memory problem, these are much more likely to have triggered a crash.

As I pointed out there is a suspiciously similar problem AR#71064 which also resulted in crashing Windows (not easy to do nowaways!).

BTW I have no idea whether Xilinx limits the number of cores in the (hobbiest) WebPack installation. I did try to search for an answer to this, since my new CPU has 16 cores. My projects are not big in terms of code size, but the time taken from an edit to get a new binary is frustratingly slow (or has been - no idea what speedup I get until I can fix the crash problem).

I would be happy to implement any recommended (limited!) settings, try again and report what happens.

migry
Observer
Observer
2,630 Views
Registered: ‎05-07-2020

Hello @watari , I like your handle! Ironically the project I am currently working on is related to an add-on for the Atari ST.

For reasons mentioned in my other reply (it is well specced commercially built PC) I don't think it is likely to be thermals. The PSU is what appears to me to be an totally overspecced Corsair 1600W model. The power is only really needed for the powerful GPU when playing games. So just idling when running the desktop and Vivado. In fact while the side panel was removed I was shocked to see that the PSU fan was not even moving! Turns out this is perfectly normal when the PSU is not heavily loaded.

Just FYI when playing games, I am able to monitor GPU temperatures (and can hear the increase in fan speed!) and all is well in acceptable range.

While Vivado was running for one time I did monitor the CPU usage and no more than 2 cores were every 100% active.

I would be happy to implement any recommended (limited!) settings, try again and report what happens.

0 Kudos
watari
Professor
Professor
2,477 Views
Registered: ‎06-16-2013

Hi @migry 

 

Did you make sure stability of memory and likelihood of memory by ex. memtest64 and/or memtest86+ ?

Also, I strongly suggest you to use latest UEFI BIOS to prevent CPU and/or DRAM issue on your PC.

And, I suggest you to change DRAM parameter instead of dram parameter on SPD on DIMM, too.

 

Would you try it ?

 

Best regards,

migry
Observer
Observer
2,454 Views
Registered: ‎05-07-2020

Thank you for the suggestion. Happy to run any system stability checks.

Downloaded and ran memtest64. Total RAM is 64Gb. Passes this test. Interestingly all 16 cores/32 threads were 100% loaded during this test.

Downloaded and ran CinebenchR23. Ram for 10 minutes (standard single pass). All 16 cores/32 threads were 100% loaded during this test. No crash or instability. Just great performance! So unlikely to be a CPU cooling issue. BTW the CPU is cooled by a quality Corsair AIO water cooler and radiator.

I have contacted customer support for the PC seller. I asked for help in diagnosing any possibly hardware faults. I will run any checks that they recommend.

Please recall that the PC is brand new, so virtually all system drivers will already be up-to-date. I am checking with customer support to check that I can install updates without impacting warranty.

I am also happy to install the latest motherboard BIOS (as soon as the PC seller OKs this). DRAM settings are all motherboard defaults. I will try to figure out if there is a safe low speed setting in order to rule out DRAM problems, but memtest64 would suggest this is unlikely to be the problem - but I could be surprised.

At the end of the day I just want to be able to do my Vivado compiles much much faster!

0 Kudos
migry
Observer
Observer
2,439 Views
Registered: ‎05-07-2020

Last night last thing before powering off the PC, I loaded Vivado, loaded a known project, restored the corrupted XPR file, and ran synthesis. The PC crashed immediately (no blue screen, just a black screen) and re-booted.

I checked Windows10 Event Viewer and saw the same "Level=Error, Source=EventLog, Event ID = 6008" entry. There did not appear to be any DUMP file. I checked settings and dump file creation is enabled, so no idea why no dump file.

I will again point out AR#71064 whereby Vivado would crash (not clear if this was a BSOD crash or black screen crash like I see). I point this out to show that my problem might actually be a software problem and not necessarily a hardware issue.

Description


there have been reports of Vivado crashing when a source file is added or when a project is opened and srcscanner is run to update the hierarchy.

Once it starts happening, the crash will occur on all versions of Vivado from the 2017.1 release on.

The problem is seen under the following situations.

Opening a project that has an HDL source
Adding an HDL source to an empty project
Creating an HDL Wrapper for a block diagram (which effectively adds an HDL source to the project).

0 Kudos
migry
Observer
Observer
2,394 Views
Registered: ‎05-07-2020

Out of desperation I totally uninstalled Vivado 2020.2 (all components).

I copied the Vivado file tree from my old PC to c:\FPGA\Xilinx\Vivado\2017.4 . A different location as compared to my old PC.

I went into the bin folder and executed the viviado.bat .

To my amazement the GUI appears.

To my even more amazement I can run synthesis, run implementation, and generate a binary file. Alleluya ! 

I have no idea why it isn't complaining about a missing license file, unless it somehow is seeing the license installed for 2020.2!

 

drjohnsmith
Teacher
Teacher
2,380 Views
Registered: ‎07-09-2009

Interesting,

I hear there have been "features" with older W10 versions, so you need the latest patches, but you seem wel up on this so I guess you have them.

  not certain which version of W10 your running, and its a get out I know, but I think officially only W10 pro is supported, not home,

          But I use home on an intel machine and no problems

Hopefully some one from Xilinx / AMD will pop in and have look a this thread.

 

You mention threads, and only seeing two used in Vivado

   there is a limit in the tools , certain parts, that happen to run the longest, can only use a max of 2 cores.

Thats why its better to get processor with fewer cores runnign fast tna many cores running slower.

 

If you want to have play  

 it has been suggested that a virtual machine such as oracle , can simulate a 2 core machine using many more cores on your PC,

    QED, a VM can be faster than the raw machine is the hypothesis.

 

 

<== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
0 Kudos
richardhead
Scholar
Scholar
2,377 Views
Registered: ‎08-01-2012

Same - im on an "older" intel machine (i7-7700k, 32gb ram) with windows 10 pro 2004, and no issues running any version of vivado.

@migry  Have you tried running the synthesis via tcl or batch mode? maybe its just the GUI causing issues?

0 Kudos
migry
Observer
Observer
2,136 Views
Registered: ‎05-07-2020

That's a very good suggestion!

I am still using the GUI, but I really ought to learn how to use the command line. I guess I have been lazy, but I am also not sure where to look to find out what command set I need to compile or simulate. 

Unfortunately I uninstalled Vivado 2020.2 so I cannot try the commands stand-alone. My suspicion is that if this bug is similar to the one I have pointed out, the problem is not with the GUI, but the GUI "helper" tools, possibly "srcscanner"?

Since sort-of(!) installing 2017.4 (i.e. copied the installation folder from the old PC to the new PC) I am able to run Vivado and finally get some work done. 

Over the past few weeks I have progressed my project, so I am happy to stick with this slightly older version.

0 Kudos
migry
Observer
Observer
2,136 Views
Registered: ‎05-07-2020

Despite my PC being very expensive (mainly due to the unnecessarily powerful GPU card) it only has Window 64 Home. Nevertheless as far as I can see there are no disadvantages verses Pro. I would be shocked if any difference between Home and Pro was causing the crash, but you never know!

When choosing the CPU I did look for a processor which had one of the highest single core scores, because I am aware the free versions of FPGA tools likely have multi-core compile disabled. I would hope that companies who pay do get multi-core support, but my limited searching on this topic only gave unclear answers. I guess I need to be thankful that Xilinx allow me to have these tools for free!

BTW there is another thread where someone has a crash in Linux, on the nearly same version of Vivado. I have no idea if this is related or just a co-incidence. 

If Vivado 2020.2 was really crashing simply by compiling on a Ryzen processor, I would imagine that the forum would be full of complaints.

I have run quite a few different stress tests (some with all cores fully loaded) and also memtest64, but the PC works 100% reliably.

0 Kudos
drjohnsmith
Teacher
Teacher
2,114 Views
Registered: ‎07-09-2009

Just to clarify

Even the paid for version of code , drops back to a limited number of cores,

   Its an inherent part of the algorithms used to synthesise,

If you ever find a mutli core synthesis algorithm , then you will make a fortune...

The only speed limit I think is the size of the design in simulation, and thats well BIG,

so don't feel hard done by over the tools, you have the same ones us who pay a fortune have .

  we can just synthesise for more of the BIGGER chips than the free one, 

 

 

 

 

 

<== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
0 Kudos
surajc
Moderator
Moderator
1,884 Views
Registered: ‎01-30-2019

Hi @migry 

Are you overclocking your ryzen 9 over the manufacturer-provided limit? 
Recently a similar issue was seen in which the vivado used to crash intermittently at any stage and the root cause was overclocking the processor. 
Once the user stopped overclocking all the crashes disappeared. 

0 Kudos
migry
Observer
Observer
1,848 Views
Registered: ‎05-07-2020

Hi @surajc , nice to hear from you.

The PC was brand new and had all the suppliers stock settings, i.e. NO overclocking whatsoever. The situation today is the same, although I have updated a few drivers and the GFX card BIOS.

Even the memory was (still is) set to the Ryzen default (i.e. non XMP, non overclocked), which I understand is a rather slow setting, even though the sticks are rated at 3600MHz. The default setting is 2100MHz??? Stress tests on all CPU cores and memory (memtest64) have resulted in NO problems.

I have had a few identical crashes when playing World of Tanks (i.e. same EventLog codes). This could likely be due to the graphics drivers for the new RTX3090 (massive overkill, but the only option that the supplier had at the time).

Running Vivado 2017.4, I have had no problems at all. In fact I have been very busy today working on some HDMI code on my Numato Mimas A7 (Artix-7 based dev board).

All in all I'm very happy with my new PC, and it really has speeded up FPGA compiles!

surajc
Moderator
Moderator
1,760 Views
Registered: ‎01-30-2019

Hi @migry 

Thanks for confirming.
Will it be possible for you to share the design with me? 
I would like to reproduce it and work at my end.

you need to create the project add all the files into it and once the hierarchy update completes , go to file -> project -> archive and send me this archive project. 
Please share your email ID so that I can send an ezmove link for sharing the archive. 

Also, 
Do you have access to a Linux machine? Could you try the project on a Linux machine? 

ALso
could you set the following parameter from TCL console and try on windows again? 
set_param general.maxThreads 1 

migry
Observer
Observer
1,708 Views
Registered: ‎05-07-2020

BTW since some suggestions were that it was a power issue with the PC, despite it having a branded 1600W PSU, I have not only run stress tests with all cores fully loaded (no issue found), but I did limit Vivado to only use a single core/thread. It still crashed.

BTW I do not think that the issue is project specific. When I ran Vivado and the GUI appeared. The crash usually happened immediately after pressing "run sythesis". No idea which process was responsible since nothing was reported in the Win10 EventLog apart from "Power Failure". 

Since then I had had 2 (or 3?) crashes of a similar kind (no BSOD message) during a game of World of Tanks. After some googling, suggestions are that it could be the RTX3090 GFX drivers. I have updated the GFX card BIOS to see if this helps, however the crashes in WoT are infrequent. 

Other than Vivado and WoT the system has been reliable. 

drjohnsmith
Teacher
Teacher
1,660 Views
Registered: ‎07-09-2009

Its not much help to you, 

    but I would suggest its the configuration of your PC, 

Both Vivado and WoT are likely to be stressing the system, 

     now the fact that other stress loads dont show same problem, is a clue as to what it is, 

 

How about removing the GPU / drivers, putting a cheaper 20 dollar card in, 

       Vivado will still run quiet well ( WoT may be not ) but it will indicate if its the GPU drivers.

 

Another sugegstion,

    get a Virtual machine set up , Oracal Virtual Box ?

       install vivado into that , and see what happens,

The aim is to isolate the Vivado from the GPU, 

 

<== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
migry
Observer
Observer
1,638 Views
Registered: ‎05-07-2020

How bizarre! Another reply which I entered yesterday has gone missing. Either that or I'm going mad! I'm sure that I checked that it had been entered. :shrug:

Could it be that my posts are still moderated before being accepted? Unless the message board software detects some keyword that causes moderation???

BTW I lost a posting to Eevblog the other day. Very strange!

In the missing post I said that I would setup a new user and under a new account I would re-install Vivado 2020.2 since I only deleted the install folder but I still have the installation EXE.

I also said that there was not much point installing Linux on a another PC, and I cannot installed (easily) on this PC, but that I could use Oracle Virtual Box to install Linux (since I already run Linux using this tool) and then install Vivado. It will take a while thanks to the really slow download.

I disagree, I do not think that either Vivado or WoT are stressing the system. Vivado 2017.4 works fine and I can't see why it would stress the system significantly less than 2020.2. WoT is very likely a BIOS or driver issue, since it is such a new card. Others also report crashes with the new RTX3090 card.

I did also think about the idea of swapping video cards, but the card in my last/old PC is water cooled and so far from easy to install. I do not want to have to buy a cheap card just to track down this problem, since I have a workaround for Vivado. I can live with the WoT issue since the crash is very infrequent, as compared to the Vivado crash which was nearly, but not always, every time I clicked "run synthesis".

I will also point once again to the text in the original post, where a Vivado helper tool (srcscanner) caused a similar crash in a different version of Vivado. Apparently the bug was in some helper library. I'm surprised that Windows could not catch the bug rather than just crash outright. 

Like many bugs it could be mx of a rare hardware fault and a rare software state that triggers the crash. IDK. :shrug:

drjohnsmith
Teacher
Teacher
1,624 Views
Registered: ‎07-09-2009

dot feel hard done by re loosing posts,

   Mine seem to go also at various times, 

   ( Wonder if Xilinx are running on a Ryzon also ?

 

I feel your pain, I've had a machine that I would swear was haunted, 

     crazy things would cause crashes, 

Never really found answer, before it was scrapped after two years. 

good luck

 

<== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
richardhead
Scholar
Scholar
1,614 Views
Registered: ‎08-01-2012

Is it possible something is corrupted in the install, or you have bad sectors on the hard disk?

We have a linux build machine that started crashing during builds randomly. Turns out the disk that vivado was installed on was broken (spinning disk), but the ssd that had the project working area on was fine.

Have you tried running vivado from a different install location?

0 Kudos
migry
Observer
Observer
1,603 Views
Registered: ‎05-07-2020

Well perhaps I now know what happened to my missing post. I just typed in a long reply and then hit the "Reply" button, but nothing happened! I pressed "Reply" again and got an error about "too many posts in one minute". FFS! I waited a few minutes and pressed "Reply", the post was still not accepted/processed. WTF!

So I am about to hit "Reply" for this post, I wonder if it will work?

migry
Observer
Observer
1,558 Views
Registered: ‎05-07-2020

Hi @richardhead , well your guess is as good as mine!

As explained, this is a new PC (arrived 10th March), and the install of the OS was mainly done by the supplier. I had to go through an initial Windows 10 setup process.  I then copied the whole of the HDD contents from my old PC to the new PC, and re-installed various packages, many of which I still had the install EXE for in the old download folder.

I thought it best to install the latest version of Vivado, so I downloaded and installed. That's when I encountered the crashes (the WoT crashes came later and they are infrequency, only 3 so far). Vivado is installed on the M2 SSD. I sort of remember doing a disk check at the time, but I can't be 100% sure, since I tried several things in an attempt to diagnose, after some google searches. Any new device, PC or whatever can have some undetected issues, but I would hope that the new SSD would be "fresh" and error free. I understand that they possibly degrade over time, but I think that's only after lots of use, and anyway circuitry is present to minimise this effect by shuffling data around?

"Have you tried running Vivado from a different install location?" Not 2020.2, but I am running 2017.4 with no problems whatsoever. As explained I simply copied the tree from my old hard disk to a new location on the M2 SSD, and it just worked, much to my surprise. No need to go through the install process! What a time saver!

Perhaps I wasn't clear, but Vivado 2020.2 did run once through to the binary creation without crashing, but it was only once. I'm not sure what this tells us. Perhaps "srcscanner" (or some other similar helper tool) didn't need to run?

I'll post back next week with the results of the re-install, and the Linux experiment.

[trying to see if this message posts this time!]

0 Kudos
drjohnsmith
Teacher
Teacher
1,520 Views
Registered: ‎07-09-2009

Can I be so bold as to say , ouch..

   copying executable files from one W10 install to another,  always a big risk, 

 When I have doe that on other programs, 

    every time I got a crash  / error I wondered , was it because I did not run the installers !

I never know what the installer did with the registry, 

     and as we all know, once that's in a bad state, then the PC is less than happy 

Its like my over clocked machine, 

   its fantastic, apart from when its not, then I wonder was it the over clocking.

At this point, my suggestion would be to re install W10, install the programs correctly , and only copy over the data files, 

   BUT , I know I would not do that, so I can't expect you to either  

I'd say with the history that we have here, and my belief that there are many other people out there with AMD chips running Vivado, 

     I'm sorry to say , IMHO its your machine set up , and we are not going to be able to find it on the forum. 

 

 

<== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
0 Kudos
migry
Observer
Observer
1,504 Views
Registered: ‎05-07-2020

So @drjohnsmith , I'm afraid you have got it wrong, and I don't exactly agree with some of your other comments. Sorry!

There are lots of posts to go through, but I did explain that the initial install of Vivado 2020.2 was a clean and full install on a new PC. This version crashed repeatedly. Perhaps I was less clear that it did once run through synthesis and binary generation, just the once. This might be a clue to the problem - IDK.

Due to frustration at the long and drawn out download and installation, just as an experiment, I executed Vivado from the new location on my hard disk, where the old 2017.4 version from my old PC hard disk had been copied over. Much to my surprise, it worked. So I simply copied the Vivado 2017.4 tree from this "BACKUP" location to a more obvious location "C:\FPGA\Xilinx\Vivado" area. To my amazement Vivado 2017.4 not only runs but thankfully does not crash. I agree this is unexpected behaviour and I was surprised that I didn't have to do a full install in order to get the right entries in the registry. Nevertheless it works, I can synthesise, implement and create a binary. I am also able to simulate. I am now doing the FPGA work I need to do, and I am sure thanks to the upgraded PC, it is faster than the old PC. Anyway although this is a non-standard way of installation, I'm not changing anything while it works because it allows me to get my work done.

I also agree that many other people will be running this version of Vivado on a Ryzen9 PC. So I agree the issue is due to a particular combination of hardware/software drivers on my PC.

But I will also point out again, that an earlier version of Vivado did have an issue with a helper EXE (srcscanner) which caused PCs to crash (apparently not all). It was a library problem. So it still could be a bug in the Xilinx software or helper EXEs or libraries, triggered under specific conditions, which I just happen to have.

0 Kudos
migry
Observer
Observer
1,483 Views
Registered: ‎05-07-2020

Hi @surajc  I have now tried one experiment for you.

I created a new user. I logged in and then installed Vivado 2020.2 from scratch for this user. Despite having (what I thought was the full) install binary, it still took hours to download and install. Sigh! I also copied my current project to an area for experimentation.

I ran Vivado 2020.2 (clicked on the icon) and got the message about this project being 2017.4, "do you want to convert to 2020.2". I clicked "yes", and the PC immediately crashed and rebooted. #Nothappy .

When the PC came back up, some things were messed up, one monitor was not active any more. To be honest I'm not keen to crash my PC any more, just in case it does cause disk corruption. There was nothing helpful in the Win10 EventLog.

BTW I checked the experimentation area, and as previously reported, the XPR file is completely corrupted.(it's full of 0 bytes).

As a bonus the installer removed the working 2017.4 area, but as explained,  I now know that I can simply copy the tree back!

[Hmmm. Another post which wasn't accepted several hours ago. Let's see now!]

0 Kudos
drjohnsmith
Teacher
Teacher
1,473 Views
Registered: ‎07-09-2009

Oh you do have problems with this PC,

Good luck fixing it, 

<== If this was helpful, please feel free to give Kudos, and close if it answers your question ==>
0 Kudos
mfreesto
Visitor
Visitor
1,091 Views
Registered: ‎05-31-2021

Hello

I am having what seems to be the same problem. I was running Vivado 2020.2 on my ‘old’ Intel I9 (9900k) happily for many months. Recently I built an AMD R9 (5900X) based Windows 10 PC.

The R9 PC runs all the usual stress tests like Cinebench R23, memtest and others very nicely (and quickly). For now, I am using the stock ‘default’ setting in Ryzen master (no overclock). So, I am pretty confident that my HW is stable.

I have a fresh install of Windows 10 with all updates applied. I also freshly downloaded and installed Vivado 2020.2.

However, the R9 PC *immediately* crashes when I try to open a known good project.  Nothing graceful at all – black screen and reboot. I tried the boostSharedDirFixApp.exe workaround but nothing changed.

Surely many others are running Vivado on AMD 5900X platforms??? Is AMD R9 a tested and supported platform within Xilinx (I hope so given that they will soon be the same company)?

Any suggestion on where to find a crash dump of some other clues would be appreciated!

Thank you

Mike

0 Kudos