03-09-2018 08:55 AM
I've run into a small FPGA team who are using cocotb. It utilizes Python for testbenches, instead of (system)verilog. I can't find much on it except a dated blog or two from the developers promoting it.
If anyone has info or experience with it please share.
Thanks,
Mike
03-09-2018 03:27 PM
Not heard of it - but looks similar to MyHDL and VUnit (although MyHDL is solds as an entire development option, with auto generation of HDL from Python).
These have been around a few years and havent really seen them getting much traction, outside a few engineers and the odd company/team. If it works for them, great. More testing, and better testing is always a good thing.
I think the main problem is the fact they are open source. They may be amazing, but the level of support is just not there like you get for VHDL and UVM. Im sure some python devs will shout about how easy it is and how its amazing, and how we're all stupid for not using it. But everyone seems fine without it?
They are up against the fact that it adds a layer of abstraction, and most FPGA engineers dont trust extra abstraction. And without proper vendors pushing it, it really wont get very far. Then as the popularity wanes, people will be confused how to use these old tools that no-one else seems to use any more.
Ill wait until something like this is pushed and supported by training providers and tool vendors.
PS. This is my opinion. I have seen respected collegues use these tools to great effect, but nothing has yet convinced me I need these in my life.
03-10-2018 01:13 PM
Cocotb can neither be compared to MyHDL nor to VUnit.
MyHDL is a hardware description language build on top of Python.
VUnit is a unit testing framework implementing an xUnit framework for VHDL (and Verilog and SystemVerilog). It uses Python as to orchestrate simulators.
Cocotb is a co-simulation framework using a native binary library written in C and C++ to attache a Python interpreter to a simulators API. This is either VPI (Verilog Programming Interface) for GHDL, VHPI (VHDL Programming Interface) for Riviera-PRO / Active-HDL or FLI (Foreign Language Interface) for ModelSim/QuestaSim. These libraries allow deep interaction with the simulation kernel to create, control and events and signal values.
The Vivado Simulator does not support such interface and thus can not be used with Cocotb. Moreover, the newest xSim will add a co-simulation interface, but does not adhere to language (VHPI, VPI) or industry standards (FLI) for simulator interfacing.
I personally, liked the idea of Cocotb, but I also consider this framework dead. Several tries to preserve the momentum by me and others have been rejected. The developers got new jobs and do not work on Cocotb anymore, nor have they assigned new developers for that project. See this discussion: https://github.com/potentialventures/cocotb/issues/513
The PoC-Library has experimented with Cocotb and implemented several advanced testbenches in Python. See this advanced cache testbench: https://github.com/VLSI-EDA/PoC/blob/master/tb/cache/cache_par_cocotb.py?ts=2
The free VHDL simulator GHDL is currently adding native Python support. This adds a native Python API, which offers new possibilities in cross language verification. Sets see what the future brings...
03-10-2018 03:51 PM
03-11-2018 12:04 AM
Cocotb can neither be compared to MyHDL nor to VUnit.
Maybe not directly because they have different targets, but they were all designed with the main goal being to improve testing, with the massive array of python libraries available to generate test vectors being cited by all.
Dont get me wrong, I am totally in favor of anything that improves testing. And VUnit is something that has come up on my radar and will be looking into more. But what you're up against is an FPGA industry that has a certain apathy towards testing.
For every big (and many small) company "doing it right", there are many more engineers in smaller companies where designs are tested by waveform and on the chip. Anyone coming to a company like this and trying to introduce something like CocoTB or Vunit will likely get penalised for "not getting things to market quick enough". They are trapped in a never ending cycle of rushed designs, patchy interfaces and a general feeling of "I dont really know how this works". Then they sell their product and move on to the next one where it all happens again. Many engineers would like to do better testing, but they often feel pressured to get the firmware on the chip, and dont have the time/skills to investigate how to do it better.
Looking through what I would consider the two larger training providers (doulos and FirstEDA), there is no mention of VUnit, CocoTB or similar. Their main courses are VHDL, SV and UVM. Without training being available, it will be hard to get the message across. I think most usages will be via "word of mouth", where someone has done the legwork, got it all set up, and set up training internally. Engineers move on and take this knowledge with them, but I feel without the training and industry support from the big 3 (mentor, Active, cadence) to keep up with an industry standard methodology there is a danger of bit rot over time as the teams using it will inevitably move on.
On a cynical note, I suspect the support is lagging because vendors/trainers do not see a money making opportunity with it. Look at HLS and OpenCL as an example. C to gates has been around for a long time, But things like Handel-C, Catapault-C and System-C were not supported natively by Xilinx/Altera and developed by 3rd party companies, and therefore pretty niche. But now Xilinx and Altera/Intel have seen the huge potential to pull software developers into their market with OpenCL and HLS, they are providing big support for them. Xilinx and Altera LOVE inefficient but easy to code designs - it sells bigger chips, and more chips.
03-11-2018 01:27 AM
Doulos is starting to offer first webinars around UVVM and is offering webinars since some time for OSVVM - a similar framework.
The companies behind UVVM and OSVVM offer trainings. It's their business model to deliver these trainings for the provided open source frameworks. Both are also VHDL verification frameworks. UVVM is on the market since 2 years; OSVVM for more then 10 years. While OSVVM is already integrated in all enterprise simulators, UVVM is now getting integrated.
03-11-2018 03:22 AM
Yes, I have seen those webinars, and I am actively using OSVVM (I have known about OSVVM since the initial posts about it on comp.lang.vhdl).
I much prefer OSVVM as it is more a toolbox, while UVVM is an entire framework (and I also think if you wanted UVM, why not just use UVM?). Something like OSVVM is much easier for a casual designer to use as they can pick and chose bits to use in their own tests, and build it up over time. Something like UVVM they have to put their DUT into a UVVM testbench, which is a lot of initial work that many cannot afford or are just put off by, and is likely a test methodogy many have not seen or used before. (and the cynic in me thinks its all just a ruse to offer training in something)
I think something as heavyweight as UVVM can suffer more from bit rot than OSVVM.
03-11-2018 07:33 AM
03-12-2018 10:55 AM
Your responses and views are very interesting and appreciated. BTW, I never found much on cocoTB except a few red flags indicating it lost what traction it may have had.
Regardless, one immediate downside is 'skill-set portability' (if you will). Any FPGA designer, who would normally leverage from their years of HDL-based testbench experience, is basically reduced to a newbie when resorting to another language/method such as cocoTB or similar. There's got to be a significant upside for it to make sense, but I'm not seeing it, and the reason for this post.
In general, when assessing new tools and methods, I think it's important to take into account the size and complexity of the design/FPGAs involved. Excuse the analogy, but like boats, FPGA designs can range from small, simple runabouts, to enormously large, complex cruise ships. Picture an 8 passenger ski boat docked along side an 8500 passenger Royal Caribbean cruiser. (For what it's worth, the analogy isn't too far off if using passenger capacity and logic cell density; the largest is about 150,000% larger then the smallest.)
With this perspective in mind, one might draw different conclusions when assessing new tools and methods...
For smaller, simpler designs, I suspect most would agree it makes more practical sense to simply develop testbenches in the same hardware description language used for RTL. Regarding abstraction, VHDL and System Verilog offer plenty without having to switch to a non hardware language like Python.
For large, complex designs especially team-based such as FPGA emulation of ASICs (with generational roadmaps offering lots of reuse opportunity), I suspect most would agree investing in something like UVM may make sense.
Vendors of course, who have an obvious agenda, will lean toward their solutions, which may or may not really make sense. Many articles and benchmarks promoting great advantages are written or influenced by vendors so follow the money.
The question in my mind is where on the size/complexity spectrum does it begin to make sense to develop testbenches in anything other then a hardware description language. And where on the spectrum does it make sense to adopt something complex like UVM.
For me personally, most of my designs have been medium sized designs (which isn't a helpful metric especially considering the boat analogy). I've always done just fine writing testbenches, even complex ones with reuse in mind, using VHDL and System Verilog. I never perceived this to be a limitation or problem needing a solution.
03-12-2018 01:41 PM
04-10-2018 03:35 PM
After just reading through the comments, I thought I'd throw in another perspective -- mostly from the ASIC development side and less so from the FPGA prototyping/emulation arena.
We are actually just starting to transition to cocotb instead of SV/UVM for testing (we'll see how it goes). There are a few motivators for doing this:
* We are trying to verify a mixed-signal design where dependencies on the analog portion of the design often requires binding a virtual interface deep into the (often changing) hierarchy to test what we want. Also, as new tests are written, they often require additional signals in the hierarchy which then requires updating the virtual interface and the binding. This list gets long. Keeping both the virtual interface and bindings up to date is a bit of a pain, and cocotb basically removes the need for the virtual interfaces -- so almost no setup/maintenance required.
* Our lab test infrastructure is based off of python, so switching to cocotb/Python would allow us to reuse tests/code between the lab and verification environments. Only low level interface functions need to be swapped out.
* Data manipulation is easier in python. So for example, in a mixed-signal simulation, when we are looking at signal sets that are not strictly digital, there are more tools at our disposal to determine if the an analog signal (that we can capture) has the proper characteristics than if we were operating entirely in SV.
* Logging results. Once we have our results, there are many more options to save the results in whatever format and location we'd like since we're in Python. This seems to allow a more natural transition to CI or other automated methodologies.
In the few trials we've run so far, I'd say it was far easier to setup the test infrastructure (from scratch) to the point where engineers can start writing tests -- this was almost trivial. I'd say writing assertions in cocotb is a bit more cumbersome or rather less compact than SV unless you create some kind of convenience class.
Our situation is probably not typical, so take my comments with a grain of salt. So far we haven't been limited by cocotb compared to what we had been doing. Perhaps we'll run into some other roadblocks as we use it more, but so far, it looks promising. I for one hope its development continues.
06-14-2018 12:03 PM
My two bit's as a cocotb user,
#2 prevents me from using a License based simulator.
GHDL/iverilog are fast compared to xsim. but do not support SV and UVM. UVVM does not work on iverilog. Also, There is hardly any documentation on UVVM
Of all the toolkits that I have explored, cocotb seems most suitable for the 2 constraints listed above.
The toolkit is usable as is today, there are a few "nice to have" features missing but no show stoppers.
Regarding long term viability of the project.
My experience with the framework is as follows:
11-14-2018 04:21 PM - edited 11-14-2018 04:21 PM
It should definitely be noted that cocotb has a new lease of life. The list of maintainers is growing.
@lionfred hits the nail on the head with his points regarding the reason for adopting cocotb. I'm particularly fond of the feature allowing one to use the same Python scripts against simulations as on the bench by simply swapping out low level transport functions.
A pity xsim doesn't support a standard interface like VPI. Their users' loss, I suppose.
11-18-2018 05:05 PM
11-18-2018 05:07 PM
@lasplund : Large companies are using VUnit for their FPGA development of high-end products shipping in million of units.
Like which large company uses it?? Any names
>> ASIC teams use VUnit for their RTL to reduce the number of bugs handed over to the verification teams where they become much more costly to debug.
Which ASIC teams? from which company, any names please.
>>Tool vendors support VUnit by providing tool provider licenses
Which vendors support? Names please
11-18-2018 06:41 PM
I'm not allowed to name big companies that use VUnit, but I can assure, I'm not Lars and the company I know is a top-ten customer of Xilinx.
Vendors supporting VUnit: e.g. Sigasi
VUnit is shipped with OSVVM, which in turn is supported by ModelSim, QuestaSim, Active-HDL, Riviera-PRO, ...
To conclude, only Xilinx does not support any of the emerging open source verification libraries. Some of them have gained almost 20% market share according to the latest Wilson Research Report about ASIC / FPGA designs.
05-07-2020 07:59 AM
I know this is an old post, however, a lot has changed with cocotb since this conversation started on the Xilinx forums. I only heard about cocotb a few years ago but again, a lot has changed since then.
First, cocotb has had a huge make over since 2018 and the number of contributors has increased significantly and changes are made daily on Github. I haven't contributed to the project but I am an active user now.
I would like to add more details on this (likes/dislikes, reason to use it, type of development being done, etc.) but this post may be too old and a waste of time so I may open up another discussion on the general forum.
05-07-2020 01:50 PM
Cocotb has certainly made rapid strides since I mentioned it last.
05-07-2020 04:46 PM
I'll third the claim that cocotb has come a long way in the past 18 months.
There's been a lot of development, there are regular releases, and the development community are very responsive and constructive.
I still use it daily in regression runs, and although I've not done a lot of work on my cocotb verification infrastructure recently, they've done a very good job in ensuring backward compatibility and I've not noticed any of my sims breaking when getting the latest version.
05-08-2020 09:08 AM - edited 05-08-2020 09:11 AM
Very good to hear all of this because the problem that it solves has been the next reckoning for FPGA/ASIC development. In case we haven't agreed on a problem statement I will state it in my point of view (and opinion):
FPGA/ASIC designers are "mostly" designing systems that expand peripherals for CPUs. When this is true the choice verification platform would be one that is most capable of emulating a CPU.
The system that best fits the aforementioned problem statement and benefits our industry the most is one that:
(Spoiler: cocotb does all of this)
I changed jobs a few years ago and at some point another engineer (firmware guy that also did FPGA development) brought up cocotb. I couldn't argue about the advantages of taking HDL to a higher level language that had such a huge community so I reluctantly agreed that it was worth investigating. I say "reluctantly" because I am a hardware engineer and I am comfortable with C and Java only and I wasn't excited about learning yet another programming language.
Fast forward to the end and after a year or two of writing more python than HDL we now have a rapid development platform that accelerates R&D deployment to a maximum. I don't want to sound over supportive or overly excited but simulators with VHPI/VPI hooks mixed with Python are the shiznit (please excuse my huggie dizzle) and this has quietly opened up a huge advancement in our industry....and nobody is talking about it!
I will add that I am a huge fan of Python now too, replacing TCL wherever I can and scripting every monotonous task that gets in my way. Who knew that software engineers had all the fun?...okay, now I am getting carried away
05-08-2020 10:13 PM
To Illustrate how cocoTB shines in #1, #2 and #3 above,
In less than a week we were able to provide our Driver Development team an environment which had.
Imagine even trying this with System Verilog!
Imagine giving your application/Driver team an executable bit accurate environment in which they can test their work while RTL development is still taking place.
For my next project I am thinking of completely ditching SystemC and doing my system modeling in Python
05-09-2020 10:21 AM
For my next project I am thinking of completely ditching SystemC and doing my system modeling in Python
This last statement you made is the reason I made a new post on an old thread. To add to it, I am modeling EVERYTHING in Python now (I've dropped Matlab too which I've relied on heavily over the years and have always battled with other engineers for licenses). Cocotb allows designers to choose either Verilog or VHDL (no system Verilog/C, or other crazy ideas). I've even dropped Vunit and OSVVM and with that I no longer have to use the clumsy, hard to read, hard to write code for opening/saving to files, using variables, and every other HDL obfuscation related to trying to write C for test benches in a language that describes hardware.
Another point you made that is huge that has quietly shown up at our doorsteps:
Imagine giving your application/Driver team an executable bit accurate environment in which they can test their work while RTL development is still taking place.
I have been running the Pynq (Python running on Zynq) framework to take advantage of this for a while (and others within my team). So, we don't have to imagine it anymore or write a bare-bone OS on a Zynq in C++ to verify....As a matter of fact, I am writing test benches that port directly to my SoC in Python (Cocotb based TB file written in Python that I can load onto my Zynq with the Pynq framework running on Xilinx's petalinux distro).
If you don't have an SoC it really doesn't matter. You can get a cheap dev board and flush out the main issues with your comm. interface and being diligent in Python you can accurately model your HDL. Imagine that
I've probably spent my last two years developing in Python and using Cocotb with Pynq and I've now gotten to a place where Research, Development, Verification and Deployment has reached an apex of rapid, reusable and robust FPGA/ASIC design from cradle to grave. Yep, that was a mouthful and I'm not in marketing...it's just, "the way"....finally.
Oh yeah, since we can use Python for all of the above, do we even need to have the discussion anymore about how under-served the tool market is for FPGA/ASIC developers? I'm using VSCode now...I am feeling spoiled because I don't have to spend my career using Notepad++, paid for editors, proprietary tools or, ahem...VIM or EMACS.
I just need to add one plug to this before I forget.....is XSIM going to ever add VHPI/VPI support?! One of the few mentions of Cocotb on the Xilinx forums.
I wish I had an @AllOfXilinx option.