cocotb. Anyone hear of it?

[gulotta]

I've run into a small FPGA team who are using cocotb.  It utilizes Python for testbenches, instead of (system)verilog.  I can't find much on it except a dated blog or two from the developers promoting it.  

If anyone has info or experience with it please share.

Thanks,

Mike

[richardhead]

Not heard of it - but looks similar to MyHDL and VUnit (although MyHDL is solds as an entire development option, with auto generation of HDL from Python).

These have been around a few years and havent really seen them getting much traction, outside a few engineers and the odd company/team. If it works for them, great. More testing, and better testing is always a good thing.

I think the main problem is the fact they are open source. They may be amazing, but the level of support is just not there like you get for VHDL and UVM. Im sure some python devs will shout about how easy it is and how its amazing, and how we're all stupid for not using it. But everyone seems fine without it?

They are up against the fact that it adds a layer of abstraction, and most FPGA engineers dont trust extra abstraction. And without proper vendors pushing it, it really wont get very far. Then as the popularity wanes, people will be confused how to use these old tools that no-one else seems to use any more. 

Ill wait until something like this is pushed and supported by training providers and tool vendors.

PS. This is my opinion. I have seen respected collegues use these tools to great effect, but nothing has yet convinced me I need these in my life.

[paebbels]

Cocotb can neither be compared to MyHDL nor to VUnit.

MyHDL is a hardware description language build on top of Python.

VUnit is a unit testing framework implementing an xUnit framework for VHDL (and Verilog and SystemVerilog). It uses Python as to orchestrate simulators.

Cocotb is a co-simulation framework using a native binary library written in C and C++ to attache a Python interpreter to a simulators API. This is either VPI (Verilog Programming Interface) for GHDL, VHPI (VHDL Programming Interface) for Riviera-PRO / Active-HDL or FLI (Foreign Language Interface) for ModelSim/QuestaSim. These libraries allow deep interaction with the simulation kernel to create, control and events and signal values.

The Vivado Simulator does not support such interface and thus can not be used with Cocotb. Moreover, the newest xSim will add a co-simulation interface, but does not adhere to language (VHPI, VPI) or industry standards (FLI) for simulator interfacing.

I personally, liked the idea of Cocotb, but I also consider this framework dead. Several tries to preserve the momentum by me and others have been rejected. The developers got new jobs and do not work on Cocotb anymore, nor have they assigned new developers for that project. See this discussion: https://github.com/potentialventures/cocotb/issues/513

The PoC-Library has experimented with Cocotb and implemented several advanced testbenches in Python. See this advanced cache testbench: https://github.com/VLSI-EDA/PoC/blob/master/tb/cache/cache_par_cocotb.py?ts=2

The free VHDL simulator GHDL is currently adding native Python support. This adds a native Python API, which offers new possibilities in cross language verification. Sets see what the future brings...

[lasplund]

They are up against the fact that it adds a layer of abstraction, and most FPGA engineers dont trust extra abstraction.

VUnit was developed with RTL developers in mind and that's is why it's based on VHDL/SystemVerilog. It supports high-level test abstractions but the basic use is much more direct.

These have been around a few years and haven't really seen them getting much traction, outside a few engineers and the odd company/team

VUnit it's not so odd these days. Large companies are using VUnit for their FPGA development of high-end products shipping in million of units. ASIC teams use VUnit for their RTL to reduce the number of bugs handed over to the verification teams where they become much more costly to debug.

Ill wait until something like this is pushed and supported by training providers and tool vendors.

Tool vendors support VUnit by providing tool provider licenses, they test their simulators with VUnit internally and they are also starting to integrate VUnit with their tools (SigasiStudio). VUnit is also used by the "provider" of VHDL. The VHDL standardization group is using VUnit to verify the VHDL standard packages.

[richardhead]

Cocotb can neither be compared to MyHDL nor to VUnit.

Maybe not directly because they have different targets, but they were all designed with the main goal being to improve testing, with the massive array of python libraries available to generate test vectors being cited by all. 

Dont get me wrong, I am totally in favor of anything that improves testing. And VUnit is something that has come up on my radar and will be looking into more. But what you're up against is an FPGA industry that has a certain apathy towards testing.

For every big (and many small) company "doing it right", there are many more engineers in smaller companies where designs are tested by waveform and on the chip. Anyone coming to a company like this and trying to introduce something like CocoTB or Vunit will likely get penalised for "not getting things to market quick enough". They are trapped in a never ending cycle of rushed designs, patchy interfaces and a general feeling of "I dont really know how this works". Then they sell their product and move on to the next one where it all happens again.  Many engineers would like to do better testing, but they often feel pressured to get the firmware on the chip, and dont have the time/skills to investigate how to do it better.

Looking through what I would consider the two larger training providers (doulos and FirstEDA), there is no mention of VUnit, CocoTB or similar. Their main courses are VHDL, SV and UVM. Without training being available, it will be hard to get the message across. I think most usages will be via "word of mouth", where someone has done the legwork, got it all set up, and set up training internally. Engineers move on and take this knowledge with them, but I feel without the training and industry support from the big 3 (mentor, Active, cadence) to keep up with an industry standard methodology there is a danger of bit rot over time as the teams using it will inevitably move on. 

On a cynical note, I suspect the support is lagging because vendors/trainers do not see a money making opportunity with it. Look at HLS and OpenCL as an example. C to gates has been around for a long time, But things like Handel-C, Catapault-C and System-C were not supported natively by Xilinx/Altera and developed by 3rd party companies, and therefore pretty niche. But now Xilinx and Altera/Intel have seen the huge potential to pull software developers into their market with OpenCL and HLS, they are providing big support for them. Xilinx and Altera LOVE inefficient but easy to code designs - it sells bigger chips, and more chips.

[paebbels]

Doulos is starting to offer first webinars around UVVM and is offering webinars since some time for OSVVM - a similar framework.

The companies behind UVVM and OSVVM offer trainings. It's their business model to deliver these trainings for the provided open source frameworks. Both are also VHDL verification frameworks. UVVM is on the market since 2 years; OSVVM for more then 10 years. While OSVVM is already integrated in all enterprise simulators, UVVM is now getting integrated.

[richardhead]

Yes, I have seen those webinars, and I am actively using OSVVM (I have known about OSVVM since the initial posts about it on comp.lang.vhdl).

I much prefer OSVVM as it is more a toolbox, while UVVM is an entire framework (and I also think if you wanted UVM, why not just use UVM?). Something like OSVVM is much easier for a casual designer to use as they can pick and chose bits to use in their own tests, and build it up over time. Something like UVVM they have to put their DUT into a UVVM testbench, which is a lot of initial work that many cannot afford or are just put off by, and is likely a test methodogy many have not seen or used before. (and the cynic in me thinks its all just a ruse to offer training in something)

I think something as heavyweight as UVVM can suffer more from bit rot than OSVVM. 

[lasplund]

Anyone coming to a company like this and trying to introduce something like CocoTB or Vunit will likely get penalised for "not getting things to market quick enough"

That's a good point and a reason why UVM is so hard to get started with. If you're an FPGA team using VHDL you have to learn both SystemVerilog and UVM. You also have to acquire new simulator licenses so it all becomes a major corporate decision. You cannot reuse testbenches you already have and it's hard to start with something trivial since UVM wasn't designed to make trivial testbenches. The VUnit approach is to start where you are and take baby steps forward. For example, if you're fast you can install Python and VUnit in a minute (https://www.linkedin.com/pulse/vunit-best-value-initial-effort-lars-asplund/). After that you can spend another minute to create an incremental compile script for your project (https://www.linkedin.com/pulse/vunit-best-value-initial-effort-part-2-lars-asplund/). Now you've started to add value. Next step is to add five lines of code to your testbenches to get everything fully automated. With a single command you can run all you testbenches and you have support for distributing the simulations on multiple CPU cores to speed up the process. More added value. At this point you haven't really changed your testbenches, you start where you are. The testbenches didn't even have to be self-checking and you don't have to replace their asserts if they were to make this work.

At this point you would go to https://vunit.github.io/index.html to figure out what the next small step is. Even though VUnit comes with a Python scripts you still have a pure RTL testbench at the core which can be simulated without the Python test runner. This means that you can start using VUnit for yourself without getting the rest of your team/company involved, all they see is RTL code. This is something we see often. Starting with VUnit is a personal decision. You would benefit from having the whole team or company using the same tool but it doesn't have to happen at once. Baby steps. Start where you are. Adapt at your preferred pace. 

[gulotta]

Your responses and views are very interesting and appreciated.  BTW, I never found much on cocoTB except a few red flags indicating it lost what traction it may have had. 

Regardless, one immediate downside is 'skill-set portability' (if you will).  Any FPGA designer, who would normally leverage from their years of HDL-based testbench experience, is basically reduced to a newbie when resorting to another language/method such as cocoTB or similar.  There's got to be a significant upside for it to make sense, but I'm not seeing it, and the reason for this post.

In general, when assessing new tools and methods, I think it's important to take into account the size and complexity of the design/FPGAs involved.  Excuse the analogy, but like boats, FPGA designs can range from small, simple runabouts, to enormously large, complex cruise ships.  Picture an 8 passenger ski boat docked along side an 8500 passenger Royal Caribbean cruiser.  (For what it's worth, the analogy isn't too far off if using passenger capacity and logic cell density; the largest is about 150,000% larger then the smallest.)

With this perspective in mind, one might draw different conclusions when assessing new tools and methods...

For smaller, simpler designs, I suspect most would agree it makes more practical sense to simply develop testbenches in the same hardware description language used for RTL.  Regarding abstraction, VHDL and System Verilog offer plenty without having to switch to a non hardware language like Python.  

For large, complex designs especially team-based such as FPGA emulation of ASICs (with generational roadmaps offering lots of reuse opportunity), I suspect most would agree investing in something like UVM may make sense.

Vendors of course, who have an obvious agenda, will lean toward their solutions, which may or may not really make sense.  Many articles and benchmarks promoting great advantages are written or influenced by vendors so follow the money.

The question in my mind is where on the size/complexity spectrum does it begin to make sense to develop testbenches in anything other then a hardware description language.  And where on the spectrum does it make sense to adopt something complex like UVM. 

For me personally, most of my designs have been medium sized designs (which isn't a helpful metric especially considering the boat analogy).   I've always done just fine writing testbenches, even complex ones with reuse in mind, using VHDL and System Verilog.  I never perceived this to be a limitation or problem needing a solution.  

[richardhead]

I think the point should be that testing is never a bad thing, however you do it. The problem is often one of time and resources. Engineers are often busy enough trying to get releases out and learning new tools when they feel comfortable with existing ones is always going to be a hard argument.

I will add, when I have worked on projects that have dedicated verification bods, it is hugely beneficial. You get two people looking at a spec from different angles - "how can I make this work" and "how can I put this design under pressure". When we had this in place, for a large project with 1/2 dedicated verification guys, the bug rate after completion did get good feedback from upper management.

So, regardless of how you test - make sure you test properly (and all methods mentioned here are good ways of testing).

[lionfred]

After just reading through the comments, I thought I'd throw in another perspective -- mostly from the ASIC development side and less so from the FPGA prototyping/emulation arena.

We are actually just starting to transition to cocotb instead of SV/UVM for testing (we'll see how it goes).  There are a few motivators for doing this:

* We are trying to verify a mixed-signal design where dependencies on the analog portion of the design often requires binding a virtual interface deep into the (often changing) hierarchy to test what we want. Also, as new tests are written, they often require additional signals in the hierarchy which then requires updating the virtual interface and the binding. This list gets long. Keeping both the virtual interface and bindings up to date is a bit of a pain, and cocotb basically removes the need for the virtual interfaces -- so almost no setup/maintenance required.

* Our lab test infrastructure is based off of python, so switching to cocotb/Python would allow us to reuse tests/code between the lab and verification environments.  Only low level interface functions need to be swapped out.

* Data manipulation is easier in python. So for example, in a mixed-signal simulation, when we are looking at signal sets that are not strictly digital, there are more tools at our disposal to determine if the an analog signal (that we can capture) has the proper characteristics than if we were operating entirely in SV.

* Logging results. Once we have our results, there are many more options to save the results in whatever format and location we'd like since we're in Python. This seems to allow a more natural transition to CI or other automated methodologies.

In the few trials we've run so far, I'd say it was far easier to setup the test infrastructure (from scratch) to the point where engineers can start writing tests -- this was almost trivial.  I'd say writing assertions in cocotb is a bit more cumbersome or rather less compact than SV unless you create some kind of convenience class.

Our situation is probably not typical, so take my comments with a grain of salt. So far we haven't been limited by cocotb  compared to what we had been doing. Perhaps we'll run into some other roadblocks as we use it more, but so far, it looks promising.  I for one hope its development continues.

[jvs@dyumnin.com]

My two bit's as a cocotb user,

1. I would like to learn one verification framework /language and use it across both VHDL and Verilog,
2. I want the option of running multiple regressions over multiple machines simultaneously.

#2 prevents me from using a License based simulator.

GHDL/iverilog are fast compared to xsim. but do not support SV and UVM. UVVM does not work on iverilog. Also, There is hardly any documentation on UVVM

Of all the toolkits that I have explored, cocotb seems most suitable for the 2 constraints listed above.
The toolkit is usable as is today, there are a few "nice to have" features missing but no show stoppers.

Regarding long term viability of the project.

1. Looking at their github page(https://github.com/potentialventures/cocotb), I see that the project is still active, checkins were made 2-3 days back.
2. Looking at the issues section I see that there are some items from 2013 which are still open, but activity on other bugs is seen.
3. Since the tool does not pay for the bread and butter of the developers, I expect the development to be slow.
4. Unless the 3rd party simulators introduce breaking changes, I expect the toolkit to work even if it is abandoned by the developers.

My experience with the framework is as follows:

1. Documentation is very sparse, I had to often look through the source code to figure out what can/should be done.
2. In my first attempt I was able to setup a complete environment within a day.  With suitable generator scripts etc. this time can be reduced further.

[morsejules]

It should definitely be noted that cocotb has a new lease of life. The list of maintainers is growing.

@lionfred hits the nail on the head with his points regarding the reason for adopting cocotb. I'm particularly fond of the feature allowing one to use the same Python scripts against simulations as on the bench by simply swapping out low level transport functions.

A pity xsim doesn't support a standard interface like VPI. Their users' loss, I suppose.

[hayk.petr]

@lasplund : Large companies are using VUnit for their FPGA development of high-end products shipping in million of units.

Like which large company uses it?? Any names

>> ASIC teams use VUnit for their RTL to reduce the number of bugs handed over to the verification teams where they become much more costly to debug.

Which ASIC teams? from which company, any names please.

>>Tool vendors support VUnit by providing tool provider licenses

Which vendors support? Names please

[paebbels]

I'm not allowed to name big companies that use VUnit, but I can assure, I'm not Lars and the company I know is a top-ten customer of Xilinx.

Vendors supporting VUnit: e.g. Sigasi

VUnit is shipped with OSVVM, which in turn is supported by ModelSim, QuestaSim, Active-HDL, Riviera-PRO, ...

To conclude, only Xilinx does not support any of the emerging open source verification libraries. Some of them have gained almost 20% market share according to the latest Wilson Research Report about ASIC / FPGA designs.

[jeff_king]

I know this is an old post, however, a lot has changed with cocotb since this conversation started on the Xilinx forums. I only heard about cocotb a few years ago but again, a lot has changed since then.

First, cocotb has had a huge make over since 2018 and the number of contributors has increased significantly and changes are made daily on Github. I haven't contributed to the project but I am an active user now. 

I would like to add more details on this (likes/dislikes, reason to use it, type of development being done, etc.) but this post may be too old and a waste of time so I may open up another discussion on the general forum.

@gulotta @paebbels @hayk.petr @morsejules jvs@dyumnin.com

[jvs@dyumnin.com]

@jeff_king ,

Cocotb has certainly made rapid strides since I mentioned it last.

• A robust community is growing around it,
• It now  supports UVM and constraint driven verification.
• It now supports the fastest simulator out there (Verilator)
• It has inspired the teams behind GHDL and Verilator to investigate natively supporting python in the simulator...
• I Know it is being used inside STMicro, BRCM and other companies...
  
  I would not be surprised if it becomes the default verification platform for startups and open source ASIC community...
  
  Note: In case someone is interested in advocating it within their organisation, and need to tick a checkbox for management,
• My company (Dyumnin Semiconductors) provides cocotb based commercial verification services to interested parties....

[morsejules]

I'll third the claim that cocotb has come a long way in the past 18 months.

There's been a lot of development, there are regular releases, and the development community are very responsive and constructive.

I still use it daily in regression runs, and although I've not done a lot of work on my cocotb verification infrastructure recently, they've done a very good job in ensuring backward compatibility and I've not noticed any of my sims breaking when getting the latest version.

[jeff_king]

Very good to hear all of this because the problem that it solves has been the next reckoning for FPGA/ASIC development. In case we haven't agreed on a problem statement I will state it in my point of view (and opinion): 

FPGA/ASIC designers are "mostly" designing systems that expand peripherals for CPUs. When this is true the choice verification platform would be one that is most capable of emulating a CPU.

The system that best fits the aforementioned problem statement and benefits our industry the most is one that:

1. Executes code like a CPU.
2. Handles ISRs.
3. Is expandable to emulate multiple processors.
4. Integrates well into modern simulator environments (most/all enterprise level simulators support VHPI/VPI).
5. Is written in a well understood and supported language that isn't dead or dying (C, C++, Python, Java, etc.)

(Spoiler: cocotb does all of this)

I changed jobs a few years ago and at some point another engineer (firmware guy that also did FPGA development) brought up cocotb. I couldn't argue about the advantages of taking HDL to a higher level language that had such a huge community so I reluctantly agreed that it was worth investigating. I say "reluctantly" because I am a hardware engineer and I am comfortable with C and Java only and I wasn't excited about learning yet another programming language. 

Fast forward to the end and after a year or two of writing more python than HDL we now have a rapid development platform that accelerates R&D deployment to a maximum. I don't want to sound over supportive or overly excited but simulators with VHPI/VPI hooks mixed with Python are the shiznit (please excuse my huggie dizzle) and this has quietly opened up a huge advancement in our industry....and nobody is talking about it!

I will add that I am a huge fan of Python now too, replacing TCL wherever I can and scripting every monotonous task that gets in my way. Who knew that software engineers had all the fun?...okay, now I am getting carried away

[jvs@dyumnin.com]

To Illustrate how cocoTB shines in #1, #2 and #3 above,

In less than a week we were able to provide our Driver Development team an environment which had.

1. A Virtual machine(QEMU) running linux on a non x86 processor.
2. Connected to Python CocoTB Environment supporting
   1. CSR Read/Writes 
   2. IRQ's
   3. DMA into System memory

Imagine even trying this with System Verilog!

Imagine giving your application/Driver team an executable bit accurate environment in which they can test their work while RTL development is still taking place.

For my next project I am thinking of completely ditching SystemC and doing my system modeling in Python

[jeff_king]

jvs@dyumnin.com 

For my next project I am thinking of completely ditching SystemC and doing my system modeling in Python

---

This last statement you made is the reason I made a new post on an old thread. To add to it, I am modeling EVERYTHING in Python now (I've dropped Matlab too which I've relied on heavily over the years and have always battled with other engineers for licenses). Cocotb allows designers to choose either Verilog or VHDL (no system Verilog/C, or other crazy ideas). I've even dropped Vunit and OSVVM and with that I no longer have to use the clumsy, hard to read, hard to write code for opening/saving to files, using variables, and every other HDL obfuscation related to trying to write C for test benches in a language that describes hardware.

Another point you made that is huge that has quietly shown up at our doorsteps:

Imagine giving your application/Driver team an executable bit accurate environment in which they can test their work while RTL development is still taking place.

---

I have been running the Pynq (Python running on Zynq) framework to take advantage of this for a while (and others within my team). So, we don't have to imagine it anymore or write a bare-bone OS on a Zynq in C++ to verify....As a matter of fact, I am writing test benches that port directly to my SoC in Python (Cocotb based TB file written in Python that I can load onto my Zynq with the Pynq framework running on Xilinx's petalinux distro).

If you don't have an SoC it really doesn't matter. You can get a cheap dev board and flush out the main issues with your comm. interface and being diligent in Python you can accurately model your HDL. Imagine that

I've probably spent my last two years developing in Python and using Cocotb with Pynq and I've now gotten to a place where Research, Development, Verification and Deployment has reached an apex of rapid, reusable and robust FPGA/ASIC design from cradle to grave. Yep, that was a mouthful and I'm not in marketing...it's just, "the way"....finally.

Oh yeah, since we can use Python for all of the above, do we even need to have the discussion anymore about how under-served the tool market is for FPGA/ASIC developers? I'm using VSCode now...I am feeling spoiled because I don't have to spend my career using Notepad++, paid for editors, proprietary tools or, ahem...VIM or EMACS.

I just need to add one plug to this before I forget.....is XSIM going to ever add VHPI/VPI support?! One of the few mentions of Cocotb on the Xilinx forums.

I wish I had an @AllOfXilinx option.