02-20-2020 05:05 AM
I have a test PCIe design working, based on the Vivado PCIe DMA/Bridge Subsystem for PCI Express v4.1 examples.
This appears to work fine, but I need to reboot the Linux system so that it can see the PCIe device with lspci. (JTAG FPGA load then linux reboot).
When previously working with Virtex-6 and PCIe I used to:
1. Have FPGA logic generate the low level PCIe reset based on a time counter driven by the incomming PCI clock from core power up..
2. Use "echo 1 > /sys/bus/pci/rescan" to get Linux to re-scan the PCI/PCIe buses.
This allowed me to initially load and reload (after removing the Linux driver) the FPGA firmware and linux driver during development without rebooting the system.
I am trying to do the same with a Kintex-UltraScale KCU105 Evaluation Platform (xcku040-ffva1156-2-e). However I don't seem to be able to drive the ip cores sys_rst_n from FPGA logic. I get errors indicating that this input needs to be driven via a PORT or IBUF. It seems there is dedicated routing from the PCIe bus connectors reset pin.
I have seen mention of clearing the "use dedicated PERST routing resources" within the Vivado IDE (I assume PCIe DMA/bridge ip core configuration) but I cannot find this in Vivado 2019.2,
Is there anyway of driving this low level PCIe reset from the FPGA logic ?
Or is there some way of getting Linux to see an FPGA PCIe interface without rebooting when programing the FPGA via JTAG (or other sources) ?
02-20-2020 08:47 AM
I am not sure that you need the reset to be fabric driven for the rescan to work. After the first rescan, you could always send an in-band hot reset to the card via the configuration interface, or toggle the Root Port "link up / link down" bit for that port specifically.
That being said, I do find the "Dedicated Route" check-box on the "Basic" page, if I select advanced in 2019.2.
Have you given the "rescan" a go with the reset being routed to the card fingers works?
02-20-2020 09:26 AM
Thanks for the reply.
Doing a rescan with "echo 1 > /sys/bus/pci/rescan" does not work, a Linux reboot does.
I can't find the "Dedicated Route" check-box on the "Basic" page. Note that this is for the "DMA/Bridge Subsystem for PCI express core" IP. The lower level "Ultrascale FPGA Gen3 integrated block for PCI express" IP does have this check box.
I will have a look if I can find a way, under Linux, of toggling the specific host PC's PCIe root ports reset line if that is possible.
I can't beleive that Xilinx haven't provided a method of reseting the PCIe from the FPGA logic though. For a dedicated PCIe card it makes sense to have a hard bus driven reset, but having the ability to control the reset is needed when developing and for systems that use an FPGA with different bitstreams for different tasks.
02-20-2020 10:03 AM
I have definitely seen folks do fabric resets, so it is possible to do.
I have attached a picture of the disable dedicated route on a KCU105, Vivado 2019.2, DMA/Bridge IP. If using the example design, you also need to disable the pin constraint.
02-20-2020 10:13 AM
Many thanks for those pictures. Totaly my fault, I completely missed the scrolling down of the options under the "Basic" tab somehow. Probably as I was looking at olsder documentation which showed them in a different place on the page.
I will try that and see if it then work. Cheers for the replies.
02-21-2020 01:51 AM
Hi Terry (lI hope you are doing well),
Last time I looked at the Xilinx's XDMA driver's code, it had a reset function that makes use of PCIe Hot Reset. This resets the entire XDMA PCIe endpoint and downstream AXI logic, assuming you make appropriate use of axi_aresetn. It's about the closest thing there is to having software reset bit in the XDMA.
To perform a Hot Reset, the driver walks the PCIe device tree to find the XDMA's upstream Bridge, and pulses the secondary reset bit in the Bridge Control register. This will clear out PCIe configuration space and MSI-X vector table (if present) in the XDMA, so it must also save and restore those before and after doing the reset.
I haven't used the Xilinx driver much, so I don't know exactly how to invoke the reset function. There might an ioctl for it or a /sys entry somewhere.
02-21-2020 06:40 AM
Many thanks for that information, I will look into that. Haven't managed to progress on the PCIe reset/startup as yet as other work has got in the way.
02-24-2020 03:34 PM
02-24-2020 10:34 PM - edited 02-24-2020 10:45 PM
Many thanks for the info. Unfortunately that requires the PCIe device to have enumerated at some time as far as I can see.
Getting a bit further, it looks like the issue may be with the commodity motherboard I am using for PCIe testing, its BIOS or more recent Linux kernels. I routed the PCIe DMA blocks reset to one of the switches on the KCU105 and the user_lnk_up to an LED. The PHY link does not come up after a FPGA PCIe PHY level reset. Moving the board to an older server, that we previously used for Virtex-6 PCIe development, and the PHY link came up fine.
I am working on the premise that the Linux or the BIOS has powered down or disabled the root complex/bridge driving the 16 lane graphics board connector I am using for these tests when it didn't find anything there at boot. I will look at the Linux PCIe root controllers setups in the background of other work. It might be that /sys/pci/rescan should power up any powered down PCIe root complex/bridge's prior to the rescan.
Have been battling the Xilinx XDMA driver from git (https://github.com/Xilinx/dma_ip_drivers) to get that to work under Fedora 31. First it doesn't support later kernels (5.0.0 on perhaps earlier) and then when you fix those issues (https://forums.xilinx.com/t5/PCIe-and-CPM/PCIe-DMA-driver-compilation-issues-in-Linux-Ubuntu-19-04/m-p/1022239) that crashes. Going back to git 8b8c70b697f049649d5fa99be9c6bc4302d89ac9 "appears" to work (https://forums.xilinx.com/t5/PCIe-and-CPM/PCIe-DMA-driver-xdma-kernel-module-crashes/m-p/1070036#M15632).
Certainly my experience of the latest Vivado, Ultrascale PCIe cores, PCIe examples and their documentation has not been great so far. Seems to have gone backwards at least in some respects.This forum seems to be good though!. But am slowly coming up to speed. I will report back when I eventually find out why the PCIe interface is not coming up in a PC warm running state. In the meantime a Linux reboot will do.
02-24-2020 10:42 PM
02-24-2020 11:07 PM
Yes, I could do those things and I may have to resort to having some sort of PCIe design in the flash and/or the Xilinx Tandem PCIe system or just warm reboot once as I am doing now. However, the warm loading of a PCIe design with "echo "1" > /sys/pci/rescan" has worked well for me in the past, simplifies things a lot and appears to still work with Linux kernel 5.4.20 at least with older motherboard hardware. I will persevere, in the background, with trying to get that method working a bit longer :)
02-24-2020 11:24 PM
02-25-2020 12:04 AM
Note that in order to get the "echo "1" > /sys/pci/rescan" system to work I have always, in the past, generated a local timed reset on the FPGA from design load/boot or'rd with the physical PCIe reset line. So when the new FPGA design starts running it resets the PCIe PHY and from that the rest of the board logic on its own.
02-25-2020 02:31 AM
I have a server-grade motherboard (Supermicro X10SAT) that actually disables a slot until the next power cycle if the system boots up and doesn't see an endpoint in that slot. This means that the only way to see an FPGA-based endpoint is to flash a bitstream into the board so that it is ready for link training by the time the BIOS starts its PCIe scan.
The above is fairly unusual behaviour for a motherboard, but you may be seeing that behaviour or some variant of it.
FPGAs probably break some of the assumptions that BIOS manufacturers make (for whom FPGAs probably aren't even on their radar). As far as they are concerned, if the BIOS scan appears to show nothing in a slot, then since the slot doesn't support hot-plugging, why keep all that slot's logic running? They might disable the slot to save some power, and that state might persist until the next warm boot (or power cycle in the case of the X10SAT).