JenkinsCI vivado synthesis fails/hangs

[ashikpoojari]

Hi

We use vivado in jenkinsCI for our nightly builds. The jobs usually take a large time(time our error) or crash in jenkins environments. Not all jobs have these problem. It affects some number of builds irrespective of different hardware builds. We have tried increasing the time and ram requirements for these builds, still we have significant number of crashes and wait times for builds. Here builds are for arty boards(small fpga). Crash report is as follows or we get make error 143( jenkins kills it due to hang). Some time after two failed tries, third one would pass. Any solutions to this issue will be helpful. If we run jobs without jenkins on the same machine, the builds are stable and complete on time.

#
# An unexpected error has occurred (11)
#
Stack:
/lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x2b07c140a4b0]
/tools/xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(HARTSWorker::waitForJob()+0x4a) [0x2b07fd96078a]
/tools/xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(HARTSWorker::mainLoop()+0x98) [0x2b07fd963af8]
/tools/xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(HARTNDb::parallelListen(char const*, char const*, char const*)+0x39) [0x2b07fcfef959]
/tools/xilinx/Vivado/2019.1/lib/lnx64.o/librdi_synth.so(+0xdd9f04) [0x2b07fd191f04]

Thanks

[hemangd]

@ashikpoojari 

So based on your notes, it looks like an intermittent crash issue. 

1. Which Vivado version are you using? 

2. Which OS version are you using?

3. In which phase of synthesis you see this crash? You can check it in runme.log of Vivado.log 

>> If we run jobs without jenkins on the same machine, the builds are stable and complete on time.

Which means it might be the compatibility issues with Jenkins and Vivado. We need to check internally about it. 

[ashikpoojari]

Hi @hemangd,

Thanks for replying.

1. Which Vivado version are you using? 
     Vivado v.2019.1 , Vivado v2018.2 (64-bit)
2. Which OS version are you using?
     Ubuntu 16.04.6 LTS
3. In which phase of synthesis you see this crash? You can check it in runme.log of Vivado.log 
    Actually the synthesis completes and it hangs.The ending of the vivado.log for hangs. Then it crashes or is killed(due to hang).

Synthesis Report Ending

---------------------------------------------------------------------------------
Finished Writing Synthesis Report : Time (s): cpu = 00:14:48 ; elapsed = 00:15:50 . Memory (MB): peak = 7694.633 ; gain = 5074.492 ; free physical = 48940 ; free virtual = 340677
---------------------------------------------------------------------------------
Synthesis finished with 0 errors, 0 critical warnings and 4710 warnings.

Strace

vivado(201670)───loader(201695)───vivado(201711)─┬─vivado(229648)───loader(229672)───parallel_synth_(229690)───{parallel_synth_}(229691)
│ │ ├─{vivado}(201712)
│ │ ├─{vivado}(201713)
│ │ └─{vivado}(229640

sudo strace -p 229691
strace: Process 229691 attached
restart_syscall(<... resuming interrupted nanosleep ...>) = 0
kill(201711, SIG_0) = 0
nanosleep({5, 0}, 0x7f4fc1a5be50) = 0
kill(201711, SIG_0) = 0
nanosleep({5, 0}, 0x7f4fc1a5be50) = 0
kill(201711, SIG_0) = 0

Thanks

Ashik

[hemangd]

@ashikpoojari 

Thank you for providing the information. 

Can you share your project to reproduce this issue at our end? If yes, then i will setup the ezmove ftp through which you can provide it. 

We also need to do Jenkins server setup.

[hemangd]

@ashikpoojari 

Would you like to provide an update based on my last thread?