11-11-2020 11:51 AM
Just to set the stage here, I'm a more technically-advanced user with roughly 10 years of Xilinx FPGA design experience. I’m using a U280 Alveo card. There seems to be much less support for U280 out in the documentation and provided software, so I’m kind of floundering.
In my design I’m looking to create some FPGA kernel modules that accelerate software functions and then send/receive data to the outside world via the QSFP ports. In this manner, I'm looking to create custom kernels that send/receive data to the QSFP ports and then report that information back to the host application. The QSFP ports would be a proprietary link, so for me I am looking for help in how to actually instantiate the MGT linkages to those connectors so that I can implement my proprietary protocol.
I’m struggling with where in my design flow do I build in the QSFP functionality. The getting started guides give me an overview of what a kernel is, but what does Xilinx recommend when it comes to the architecture of partitioning functionality across kernel modules that share memory-mapped resources? How to kernels communicate to other kernels and what are the recommendations there? When should you develop a kernel vs. when should you try to build your own custom Vivado board files? There are a few AR’s that suggest you can create a custom vivado flow, but they don’t provide detail on how do you go about doing that and how that custom board flow links back in with XRT.
Any guidance is very much appreciated
11-13-2020 10:39 AM - edited 11-13-2020 10:46 AM
Hi @cohenrich ,
Thank you for providing all of the background on what you are looking at doing. There are a couple different ways to implement this system:
1. This could be done in Vitis using RTL kernels with the provided shell for the U280 (xdma_201920_3). In this scenario, the shell would have the DMA engine that communicates with the host via XRT as well as the CMS subsystem to communicate to the Satellite Controller. It would be similar to a design like this:
The shell is that black box for communication, it is described in more detail on the XRT github (https://xilinx.github.io/XRT/master/html/index.html). A short description of the loading of the platform/shell can be found here (the two stage platform doesn't apply to the u280):
If you are familiar with DFx (https://www.xilinx.com/products/design-tools/vivado/implementation/dynamic-function-exchange.html), this is essentially what's happening with the shell and user loading in their function.
In this case communication between RTL kernels would be setup through the host application/OpenCL, examples are found here:
2. This could be done in a custom/Vivado flow. This is essentially a chip down design with a provided PCB implementation. There is a CMS subsystem IP block that could be implemented to handle the communication with the SC for monitoring the board state (power/temp). There isn't an XRT DMA engine IP block, so the typical XDMA/QDMA IP would need be implemented inside the FPGA for PCIe with the respective Xilinx XDMA/QDMA driver. To your last comment, this custom board flow wouldn't link back to XRT.
The details are limited on this flow as it does involved a more in-depth understanding of working with FPGAs, more details are found on the Alveo Vivado Lounge which you can request access here (including the design constraints and the board files):
You can bring the board back into the XRT framework from the custom flow by reverting the card to it's golden image which is described about halfway down in this AR:
The decision between which is the best option will mostly depend on how close the prebuilt shell that handles all of the ports and board communications is to what your optimal solution is. Development time is also a consideration, in which you may want to focus more time working on the kernel(s).
I hope this gives a little more background on the paths to go down.
11-13-2020 11:10 AM
John - Excellent, excellent information. Thank you for all of these links. A lot of these I'd found already, but your explanation is helping tie it all together.
One question abotu the XDMA shell provided through XRT, there are many reports on forums and ARs of people having issues using AXI Stream ports with the Alveo U280. From what I can surmise, the issue is because only the QDMA core (and thus the QDMA shell) provides AXI Streaming support. Can you confirm/deny that claim? Some of the forum posts about this are quite old. I was able to gain access to the QDMA Shell Early Access Site but there's no support yet for U280 QDMA. Do you know if QDMA shell is in any roadmap for support with U280 or if it's possible to use any of those other shells at all?
11-13-2020 11:29 AM
Hi @cohenrich ,
No problem, I'm glad the post was helpful.
You're understanding is correct, the XDMA shell only supports memory mapped transfers through global memory and not through a streaming configuration. The QDMA shell does provide streaming support, but as you noted is not available for the U280 and there currently isn't a plan to support it on the U280. Unfortunately, the U200/U250 QDMA shells will not work on the U280.
If a streaming solution is required for your system, this would need to be implemented in a custom flow.