cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
spajas
Observer
Observer
3,023 Views
Registered: ‎08-20-2018

[Vivado 2019.1] Abnormal program termination (11) during synthesis

Jump to solution

My colleagues and I get this error in Vivado 2018.3/2019.1 for different projects. It is frustrating because some of our designs take 1.5 to 2u of synthesis (UltraScale+) so you can imagine how it feels when it crashes.

We all use Ubuntu 2018.04 x64 with the latest updates.

 

spajas@LinuxDev:~$ uname -a
Linux LinuxDev 4.18.0-25-generic #26~18.04.1-Ubuntu SMP Thu Jun 27 07:28:31 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
spajas@LinuxDev:~$ 

Our systems:

128GB RAM
Water-cooled Intel 9900X processor
Nvidia graphics card

Vivado is installed with 'sudo'. My project worked fine but after changing a few lines of VHDL code (can't remember which ones), it started to crash. Some of the changes I remembered so I reverted those but it didn't bring much.

Vivado synthesis was at this stage:

---------------------------------------------------------------------------------
Start Preparing Guide Design
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
Finished Doing Graph Differ : Time (s): cpu = 00:01:14 ; elapsed = 00:01:00 . Memory (MB): peak = 4083.730 ; gain = 1571.199 ; free physical = 108713 ; free virtual = 122008
---------------------------------------------------------------------------------
Abnormal program termination (11)
Please check '/home/spajas/fpga_dev/shprm13_ancore2/workspace/shprm1-nbx-fpga-vup13p/shprm1-nbx-fpga-vup13p.runs/synth_1/hs_err_pid12103.log' for details
Parent process (pid 12103) has died. This helper process will now exit

And the crash-log:

#
# An unexpected error has occurred (11)
#
Stack:
/lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7fe6ae17cf20]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(NPinC::realModule() const+0x4) [0x7fe671c5deb4]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(NPinC::design() const+0x9) [0x7fe671c5df99]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(TNPinC::tDestroy()+0xe) [0x7fe671cfdebe]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(TNInstC::tDestroy()+0xb0) [0x7fe671cf1510]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(+0x1dccf7e) [0x7fe671c59f7e]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(NBaseModC::_delInstance(NInstC*, bool)+0x1c) [0x7fe671c5250c]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(NBaseModC::destroy()+0x110) [0x7fe671c52730]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(NRealModC::~NRealModC()+0x7f) [0x7fe671c528df]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(NRealModC::~NRealModC()+0x9) [0x7fe671c52c19]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(NDesC::delModules(NRealModC*, bool)+0x148) [0x7fe671bf9b58]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(NDesC::delAllModules(bool)+0xaf) [0x7fe671bfa05f]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(NDesC::destroy(NDupStraps*)+0x357) [0x7fe671bfa8c7]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(NDbC::delDesign(NDesC*)+0x38) [0x7fe671be5d58]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(+0xc2cba1) [0x7fe670ab9ba1]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(HARTNDb::doGenomeDependencyComputation()+0x1ff) [0x7fe670abb26f]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(+0xd16933) [0x7fe670ba3933]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(+0xcaf7fd) [0x7fe670b3c7fd]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x334af) [0x7fe6a97d44af]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x34b38) [0x7fe6a97d5b38]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(Tcl_EvalEx+0x13) [0x7fe6a97d60a3]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x1e0) [0x7fe6a97d62c0]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x3b375) [0x7fe6a97dc375]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x334af) [0x7fe6a97d44af]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x76875) [0x7fe6a9817875]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x7e029) [0x7fe6a981f029]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x76) [0x7fe6a97d6156]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_commontasks.so(+0x1ebc27) [0x7fe6a3a3cc27]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x334af) [0x7fe6a97d44af]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x76875) [0x7fe6a9817875]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(TclObjInterpProcCore+0x420) [0x7fe6a9858df0]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x334af) [0x7fe6a97d44af]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x76875) [0x7fe6a9817875]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(TclObjInterpProcCore+0x420) [0x7fe6a9858df0]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x334af) [0x7fe6a97d44af]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x34b38) [0x7fe6a97d5b38]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(Tcl_EvalEx+0x13) [0x7fe6a97d60a3]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(Tcl_FSEvalFileEx+0x1da) [0x7fe6a983ac5a]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_commontasks.so(+0x208e5d) [0x7fe6a3a59e5d]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_common.so(+0x8068e2) [0x7fe6af4e48e2]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x334af) [0x7fe6a97d44af]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x76875) [0x7fe6a9817875]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x7e029) [0x7fe6a981f029]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x76) [0x7fe6a97d6156]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0xb7557) [0x7fe6a9858557]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x334af) [0x7fe6a97d44af]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x76875) [0x7fe6a9817875]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x7e029) [0x7fe6a981f029]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x76) [0x7fe6a97d6156]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_commontasks.so(+0x1ebcc4) [0x7fe6a3a3ccc4]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x334af) [0x7fe6a97d44af]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x34b38) [0x7fe6a97d5b38]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0xad486) [0x7fe6a984e486]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x346c5) [0x7fe6a97d56c5]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(Tcl_EvalEx+0x13) [0x7fe6a97d60a3]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(Tcl_FSEvalFileEx+0x1da) [0x7fe6a983ac5a]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_commontasks.so(+0x208e5d) [0x7fe6a3a59e5d]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_common.so(+0x8068e2) [0x7fe6af4e48e2]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x334af) [0x7fe6a97d44af]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(Tcl_EvalObjv+0x32) [0x7fe6a97d45e2]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x322) [0x7fe6a97d6402]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_common.so(hdi::tcltasks::task_manager::eval_in_tcl(Tcl_Interp*, Tcl_Obj*, char const*, bool)+0xab9) [0x7fe6af4e6719]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_designutils.so(HRTInvoker::inProcessLaunch(HDGUIStatus&)+0x457) [0x7fe696c0bf07]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_designutils.so(HRTInvoker::launch(HDGUIStatus&)+0x535) [0x7fe696c0cbf5]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_vivadotasks.so(+0x54c087) [0x7fe669838087]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_vivadotasks.so(+0x557095) [0x7fe669843095]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_common.so(+0x8068e2) [0x7fe6af4e48e2]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x334af) [0x7fe6a97d44af]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x34b38) [0x7fe6a97d5b38]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(Tcl_EvalEx+0x13) [0x7fe6a97d60a3]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(Tcl_FSEvalFileEx+0x1da) [0x7fe6a983ac5a]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_commontasks.so(+0x208e5d) [0x7fe6a3a59e5d]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_common.so(+0x8068e2) [0x7fe6af4e48e2]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x334af) [0x7fe6a97d44af]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(Tcl_EvalObjv+0x32) [0x7fe6a97d45e2]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x322) [0x7fe6a97d6402]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_commontasks.so(+0x23ab2f) [0x7fe6a3a8bb2f]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_commontasks.so(+0x23c1fe) [0x7fe6a3a8d1fe]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_common.so(+0x8068e2) [0x7fe6af4e48e2]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(+0x334af) [0x7fe6a97d44af]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(Tcl_EvalObjv+0x32) [0x7fe6a97d45e2]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(TclEvalObjEx+0x322) [0x7fe6a97d6402]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_commonmain.so(+0x7424) [0x7fe6aeada424]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/libtcl8.5.so(Tcl_Main+0x1d0) [0x7fe6a9841210]
/opt/Xilinx/Vivado/2019.1/lib/lnx64.o/librdi_common.so(+0x84ac9b) [0x7fe6af528c9b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fe6adf266db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fe6ae25f88f]

We observed that Vivado is (much) faster when executed in Linux than in Windows. In our case, we could save one or two hours per bitstream generation run. That is why all our workstations are switched from Windows 10 to Linux. However, I must admit I start to regret this change as this brings a lot of frustration with my collegues (and myself).

Could someone please help us/me so find a workaround or better, to resolve this issue? I am going to try the same project in Windows 10.

Thanks!

 

0 Kudos
1 Solution

Accepted Solutions
spajas
Observer
Observer
2,887 Views
Registered: ‎08-20-2018

@hemangdI was able to reproduce (11) again so I uploaded my design yesterday via ezmove FTP. Are you able to reproduce as well? Workaround would be to set "flatten hierarchy" to 'rebuilt'.

View solution in original post

0 Kudos
13 Replies
surajc
Xilinx Employee
Xilinx Employee
3,002 Views
Registered: ‎01-30-2019

Hi @spajas 

https://forums.xilinx.com/t5/Design-and-Debug-Techniques-Blog/Vivado-Synthesis-Crash-Debugging-Guide/ba-p/946862

please have a look of the above debugging a synthesis crash blog and provide us with the part of code causing crash.

Regards

Suraj C

0 Kudos
hemangd
Moderator
Moderator
2,982 Views
Registered: ‎03-16-2017

Hi @spajas , 

Tool should not crash in anyways, Can you provide your archived testcase so we can reproduce it at our end and check for a workaround and development can work on a fix for next Vivado version. 

 

I have sent you ezmove ftp through which you can provide the archived testcase. 

Regards,
hemangd

Don't forget to give kudos and mark it as accepted solution if your issue gets resolved.
0 Kudos
spajas
Observer
Observer
2,972 Views
Registered: ‎08-20-2018

@surajcI sent you a PM.

I disabled flatten_hierarchy (set to 'none') and Vivado synthesis continued. However, that is not the setting I want for my final design.

Disabling my constraints didn't work btw. I also did not try "black boxing" it yet to find the culprit.

@hemangdI will first try to reproduce it at my collegue's PC (same configuration). If it also occurs at his PC, I will TAR my design and use you link to transfer it to you. I assume all sources stay confidential?

0 Kudos
hemangd
Moderator
Moderator
2,966 Views
Registered: ‎03-16-2017

Hi @spajas , 

>>will first try to reproduce it at my collegue's PC (same configuration). If it also occurs at his PC, I will TAR my design and use you link to transfer it to you. I assume all sources stay confidential?

Yes, absolutely. 

 

Regards,
hemangd

Don't forget to give kudos and mark it as accepted solution if your issue gets resolved.
0 Kudos
hemangd
Moderator
Moderator
2,903 Views
Registered: ‎03-16-2017

Hi @spajas ,

Close this thread by marking it as accetped solution since this issue is no longer seen by you and CR has been created on abnormal program termination (6) based on the SR you have created. 

Regards,
hemangd

Don't forget to give kudos and mark it as accepted solution if your issue gets resolved.
0 Kudos
spajas
Observer
Observer
2,888 Views
Registered: ‎08-20-2018

@hemangdI was able to reproduce (11) again so I uploaded my design yesterday via ezmove FTP. Are you able to reproduce as well? Workaround would be to set "flatten hierarchy" to 'rebuilt'.

View solution in original post

0 Kudos
jfrye_wmi
Observer
Observer
2,401 Views
Registered: ‎08-19-2019

I am having this same problem. Setting "flattten hierarchy" to "none" did not fix it. Switching it back did not fix it.

Is there any indication as to why this problem occurs. What is the fix?

0 Kudos
pulim
Xilinx Employee
Xilinx Employee
2,365 Views
Registered: ‎02-16-2014

Hi @jfrye_wmi 

 

Is it possible for you to share crash dump and vivado log file to see at which stage of Vivado synthesis this crash is happening?

Vivado 2019.2 is avaiable now, can you cross-check in this version if this crash is already fixed?

If you can share testcase that will be helpful to debug this issue and figure out root cause for this crash.

 

Thanks,

Manusha

 

0 Kudos
pedro_uno
Advisor
Advisor
2,120 Views
Registered: ‎02-12-2013

I frequently get this crash during simulation.

"Abnormal program termination (11)"

That makes me think it is something in the Vivado GUI itself and not necessarily related to synthesis.

----------------------------------------
DSP in hardware and software
-----------------------------------------
0 Kudos
viviany
Xilinx Employee
Xilinx Employee
2,069 Views
Registered: ‎05-14-2008

This error indicates a crash in the tool.

There are different problems that can lead to a crash.

And the crash can happen in various processes, like synthesis, simulation, implementation, and etc.

So we need to resolve a crash case by case.

-vivian

-------------------------------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------------------------------
如果提供的信息能解决您的问题,请标记为“接受为解决方案”。
如果您认为帖子有帮助,请点击“奖励”。谢谢!
-------------------------------------------------------------------------------------------------
0 Kudos
pedro_uno
Advisor
Advisor
2,019 Views
Registered: ‎02-12-2013

 

My environment:

- Vivado 2019.2 (Vitis)

- Ubuntu 18.04 LTS

- Intel i9-9900K, 32GB ddr, nvme ssd

It happens to me in large long Ultrascale+ Verilog simulation runs. Let me know if I can help.

----------------------------------------
DSP in hardware and software
-----------------------------------------
0 Kudos
1,403 Views
Registered: ‎11-14-2019

I also get abnormal program termination (running vivado 2019.1 on linux redhat 7.8)
I have checked the guide settings (none of witch helped), but i know what lines i have to disable in my code to get things up and running.
I can boil it down to an integer calculation that sometimes (with some specific inputs) will work, but mostly doesnt
ie. i have two similar functions, which causes the termination, the other is just the opposite (lo_count) and does the exact same operation with different numbers.

function hi_count(hi_proportion: integer; lo_proportion : integer) return integer is
  variable count : integer;
begin
  count := MAX_COUNT when (lo_proportion = 0) else
    --min_count(hi_proportion, lo_proportion)*hi_proportion/lo_proportion when (hi_proportion> lo_proportion) else
    --min_count(hi_proportion, lo_proportion);
    (MIN_CYCLES*hi_proportion)/lo_proportion when (hi_proportion > lo_proportion) else
    MIN_CYCLES;
  return count;
end function;

--"High proportion " is between 0 and 127. "Low proportion" is (127- High proportion), max count is 2^20-1

So, lets say i disable the line (MIN_CYCLES*hi_proportion)/lo_proportion when (hi_proportion > lo_proportion) else
then there is no abnormal termination, but i need this calculation.

If i use the two disabled lines, in place of the two lines below (my initial system), varying inputs from the function min_count would work or not. That is, i can put in a formula that would vary the input between say for example 200 and 600 (in stead of MIN_CYCLES), but using MIN_CYCLES of 200 or 600 will cause abnormal termination. Changing to 32600 for MIN_CYCLES, and synthesis may complete if i only do this for the hi_count (if i use lo_count which switches high and low proportions in all calculations, i need a different number for MIN_CYCLES...)

0 Kudos
1,368 Views
Registered: ‎11-14-2019

I found a work around for my own problem.
Now the error I have encountered seems to be related to the integer range, as i was able to do all the calculations that caused the crash, by constraining the range of integers included in the calculation (MIN_CYCLES*hi_proportion)/lo_proportion

by defining the range of "hi_proportion" and "lo_proportion" to 7 bits (since that is their precision) in a subtype
subtype proportion_type is integer range 0 to MAX_DUTY;
...
signal hi_proportion, lo_proportion : proportion_type;

synthesis could complete without bugs. 
Thus it may seem that there is some sort of routine crashing when the synthesizer detects some oddity when it comes to performing integer division with an unconstrained integer. If there is some routine that in some occations are able to constrain the integer size, that may have been what actually made the "variable" MIN_CYCLES work in some occations. 

Anyway, this should probably either cause an error or warning, rather than crash. But for my part it does seem constraining the integer (which seems reasonable to do) is a valid workaround, although i have not tried to constrain it to +-2^31.

0 Kudos