cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
muravin
Scholar
Scholar
12,629 Views
Registered: ‎11-21-2013

managing the number of the VIVADO jobs (server loading)

Jump to solution

Hello All,

 

I am trrying to find a way to script in TCL the monitoring of the VIVADO jobs running on our FPGA-dedicated server.

 

The way we use the VIVADO business is as following: we have a relatively powerful Xenon-based server with 64GB of memory running Linux x64, we are currently on the VIVADO 2015.2, project TCL mode (gui is only used for some canvas connection updates). Everyone logs into that server through VNC session, and we do both development and synthesis runs on it.

 

Our team has recently grew a little bit and we are starting to notice effects of that. Especially, when there are multiple projects (3+ at the same time) running synthesis, which in turn is multiple IPs OOC. For multiple Implementation runs, we are ok. The effect is that we get some windows freezes for a few minutes, until these multiple OOC IP synthesis are complete.

 

What I am looking for is for a way to use TCL script to monitor the number of the jobs the VIVADO has on this server. Any suggestions are welcome.

 

Thank you

Vlad

Vladislav Muravin
0 Kudos
1 Solution

Accepted Solutions
muravin
Scholar
Scholar
21,908 Views
Registered: ‎11-21-2013
Thank you all for the responses
Vladislav Muravin

View solution in original post

0 Kudos
7 Replies
patocarr
Teacher
Teacher
12,484 Views
Registered: ‎01-28-2008

Hi @muravin,

 

  I have a very similar setup as yours, only with more servers doing the work. What I would do in your case, from Tcl do a system call to the "ps" linux command and filter it, and then parse it back in Tcl. For example:

 

Tcl:

  exec ps -eo {%C %c} | grep vivado | sort -r

 

  This will yield a list of processes running Vivado with processor usage, such as:

 

 6.8 vivado
 4.8 vivado
 4.6 vivado
 2.3 vivado
 1.8 vivado
 0.0 vivado

 

  set psout [exec ps -eo {%C %c} | grep vivado | sort -r]

 

  You can then parse it in Tcl using a loop and lindex etc.

 

Hope that helps

-Pat

 

Give kudos if helpful. Accept as solution if it solves your problem.
https://tuxengineering.com/blog

0 Kudos
muravin
Scholar
Scholar
12,450 Views
Registered: ‎11-21-2013
Thank you Pat. Yes, I am leveraging this command for now, I just thought that there is a way to do it from the VIVADO TCL.

If I may ask, how do you set up more servers to share the load? Are you using any workflow management tool? I am currently talking to our IT person to add another server so that it would share the loading.

BR
Vlad
Vladislav Muravin
0 Kudos
markcurry
Scholar
Scholar
12,445 Views
Registered: ‎09-16-2009

 

We use Sun Grid Engine (SGE) to manage our server loading.  A quick look at the wiki shows this software has gone through many variations with the acquisition by Oracle and then various other forks.

 

In any case the version we installed back in 2008 seems to be still working fine.

 

As to Xilinx integration, SGE was WELL integrated in ISE - smartxplorer was able to distribute all of it's jobs to the load servers using SGE (LSF support was availble too).  It appears that this isn't the case with Vivado.  We still use SGE with Vivado, but just to distribute multiple FPGA nightly builds.

 

It'd be nice if there was some more integration of load server software into Vivado - these FPGA are only getting bigger - and with that larger build times!

 

--Mark

 

0 Kudos
patocarr
Teacher
Teacher
12,421 Views
Registered: ‎01-28-2008

@muravin

 

  At the moment we're not doing much in terms of sharing the hosts, other than dedicating hosts for certain users. The remote hosts capability in Vivado is able to start jobs on different hosts, but only when you schedule different simultaneous runs. If you have LSF, it would be possible to run the job on an available host, though I don't have access to it to test. There's an open source LSF fork called OpenLava that looks interesting. The ideal would be for Vivado to share a job between many hosts but as far as I know, this is not possible.

 

Thanks,

-Pat

Give kudos if helpful. Accept as solution if it solves your problem.
https://tuxengineering.com/blog

0 Kudos
muravin
Scholar
Scholar
21,909 Views
Registered: ‎11-21-2013
Thank you all for the responses
Vladislav Muravin

View solution in original post

0 Kudos
11,051 Views
Registered: ‎08-14-2008
Mark, what is the command you use to launch the SGE tasks? WHen I specify to do a synthesis using LSF, it defaults to bsub. Kendrick
0 Kudos
markcurry
Scholar
Scholar
11,046 Views
Registered: ‎09-16-2009

We use non-project mode tcl batch flows.  So, it's nothing internal to Vivado at all, rather the "normal" process to start a job with SGE:

 

qsub  -b yes -j yes -cwd -sync yes -V vivado -mode batch -nojournal -source foo.tcl

Regards,

 

Mark

0 Kudos