cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
504 Views
Registered: ‎02-17-2020

PCIe block reset for Linux PCIe development

Hi,

I have a test PCIe design working, based on the Vivado PCIe DMA/Bridge Subsystem for PCI Express v4.1 examples.

This appears to work fine, but I need to reboot the Linux system so that it can see the PCIe device with lspci. (JTAG FPGA load then linux reboot).

When previously working with Virtex-6 and PCIe I used to:

1. Have FPGA logic generate the low level PCIe reset based on a time counter driven by the incomming PCI clock from core power up..

2. Use "echo 1 > /sys/bus/pci/rescan" to get Linux to re-scan the PCI/PCIe buses.

This allowed me to initially load and reload (after removing the Linux driver) the FPGA firmware and linux driver during development without rebooting the system.

I am trying to do the same with a Kintex-UltraScale KCU105 Evaluation Platform (xcku040-ffva1156-2-e). However I don't seem to be able to drive the ip cores sys_rst_n from FPGA logic. I get errors indicating that this input needs to be driven via a PORT or IBUF. It seems there is dedicated routing from the PCIe bus connectors reset pin.

I have seen mention of clearing the "use dedicated PERST routing resources" within the Vivado IDE (I assume PCIe DMA/bridge ip core configuration) but I cannot find this in Vivado 2019.2,

Is there anyway of driving this low level PCIe reset from the FPGA logic ?

Or is there some way of getting Linux to see an FPGA PCIe interface without rebooting when programing the FPGA via JTAG (or other sources) ?

 

0 Kudos
13 Replies
Highlighted
Xilinx Employee
Xilinx Employee
480 Views
Registered: ‎12-10-2013

Re: PCIe block reset for Linux PCIe development

Hi terry@beam.ltd.uk 

I am not sure that you need the reset to be fabric driven for the rescan to work.  After the first rescan, you could always send an in-band hot reset to the card via the configuration interface, or toggle the Root Port "link up / link down" bit for that port specifically.

That being said, I do find the "Dedicated Route" check-box on the "Basic" page, if I select advanced in 2019.2. 

Have you given the "rescan" a go with the reset being routed to the card fingers works?

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Highlighted
473 Views
Registered: ‎02-17-2020

Re: PCIe block reset for Linux PCIe development

Thanks for the reply.

Doing a rescan with "echo 1 > /sys/bus/pci/rescan" does not work, a Linux reboot does.

I can't find the "Dedicated Route" check-box on the "Basic" page. Note that this is for the "DMA/Bridge Subsystem for PCI express core" IP. The lower level "Ultrascale FPGA Gen3 integrated block for PCI express" IP does have this check box.

I will have a look if I can find a way, under Linux, of toggling the specific host PC's PCIe root ports reset line if that is possible.

I can't beleive that Xilinx haven't provided a method of reseting the PCIe from the FPGA logic though. For a dedicated PCIe card it makes sense to have a hard bus driven reset, but having the ability to control the reset is needed when developing and for systems that use an FPGA with different bitstreams for different tasks.

0 Kudos
Highlighted
Xilinx Employee
Xilinx Employee
459 Views
Registered: ‎12-10-2013

Re: PCIe block reset for Linux PCIe development

Hi Terry,

I have definitely seen folks do fabric resets, so it is possible to do. 

forum_PERST.PNG

 

I have attached a picture of the disable dedicated route on a KCU105, Vivado 2019.2, DMA/Bridge IP.  If using the example design, you also need to disable the pin constraint.

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
forum_PERST.PNG
0 Kudos
Highlighted
454 Views
Registered: ‎02-17-2020

Re: PCIe block reset for Linux PCIe development

Many thanks for those pictures. Totaly my fault, I completely missed the scrolling down of the options under the "Basic" tab somehow. Probably as I was looking at olsder documentation which showed them in a different place on the page.

I will try that and see if it then work. Cheers for the replies.

0 Kudos
Highlighted
Participant
Participant
366 Views
Registered: ‎12-06-2011

Re: PCIe block reset for Linux PCIe development

Hi Terry (lI hope you are doing well),

Last time I looked at the Xilinx's XDMA driver's code, it had a reset function that makes use of PCIe Hot Reset. This resets the entire XDMA PCIe endpoint and downstream AXI logic, assuming you make appropriate use of axi_aresetn. It's about the closest thing there is to having software reset bit in the XDMA.

To perform a Hot Reset, the driver walks the PCIe device tree to find the XDMA's upstream Bridge, and pulses the secondary reset bit in the Bridge Control register. This will clear out PCIe configuration space and MSI-X vector table (if present) in the XDMA, so it must also save and restore those before and after doing the reset.

I haven't used the Xilinx driver much, so I don't know exactly how to invoke the reset function. There might an ioctl for it or a /sys entry somewhere.

 

0 Kudos
Highlighted
336 Views
Registered: ‎02-17-2020

Re: PCIe block reset for Linux PCIe development

Many thanks for that information, I will look into that. Haven't managed to progress on the PCIe reset/startup as yet as other work has got in the way.

0 Kudos
Highlighted
Explorer
Explorer
291 Views
Registered: ‎08-14-2013

Re: PCIe block reset for Linux PCIe development

0 Kudos
Highlighted
270 Views
Registered: ‎02-17-2020

Re: PCIe block reset for Linux PCIe development

Many thanks for the info. Unfortunately that requires the PCIe device to have enumerated at some time as far as I can see.

Getting a bit further, it looks like the issue may be with the commodity motherboard I am using for PCIe testing, its BIOS or more recent Linux kernels. I routed the PCIe DMA blocks reset to one of the switches on the KCU105 and the user_lnk_up to an LED. The PHY link does not come up after a FPGA PCIe PHY level reset. Moving the board to an older server, that we previously used for Virtex-6 PCIe development, and the PHY link came up fine.

I am working on the premise that the Linux or the BIOS has powered down or disabled the root complex/bridge driving the 16 lane graphics board connector I am using for these tests when it didn't find anything there at boot. I will look at the Linux PCIe root controllers setups in the background of other work. It might be that /sys/pci/rescan should power up any powered down PCIe root complex/bridge's prior to the rescan.

Have been battling the Xilinx XDMA driver from git (https://github.com/Xilinx/dma_ip_drivers) to get that to work under Fedora 31. First it doesn't support later kernels (5.0.0 on perhaps earlier) and then when you fix those issues (https://forums.xilinx.com/t5/PCIe-and-CPM/PCIe-DMA-driver-compilation-issues-in-Linux-Ubuntu-19-04/m-p/1022239) that crashes. Going back to git 8b8c70b697f049649d5fa99be9c6bc4302d89ac9 "appears" to work (https://forums.xilinx.com/t5/PCIe-and-CPM/PCIe-DMA-driver-xdma-kernel-module-crashes/m-p/1070036#M15632).

Certainly my experience of the latest Vivado, Ultrascale PCIe cores, PCIe examples and their documentation has not been great so far. Seems to have gone backwards at least in some respects.This forum seems to be good though!. But am slowly coming up to speed. I will report back when I eventually find out why the PCIe interface is not coming up in a PC warm running state. In the meantime a Linux reboot will do.

0 Kudos
Highlighted
Explorer
Explorer
268 Views
Registered: ‎08-14-2013

Re: PCIe block reset for Linux PCIe development

Well, you can rip out that part of the script and point it at the switch port directly. The procedure does not require that the card be enumerated on the bus as it only involves flipping bits in the switch port upstream of the device. However, the card likely won't enumerate after boot unless you make provisions for PCIe hot plugging. This is due to how device BARs are assigned in PCIe. Unfortunately, if the card doesn't enumerate on boot, you're kinda out of luck, even if you do get the link to come up via a hot reset or similar. Hot reset is mainly useful for reloading the design without having to reboot the machine. One thing I did notice on at least one machine is that once some part of the system made a decision that there was no card in a particular PCIe slot, the slot was never checked again, even during a warm reboot. I had to physically pull the power plug and plug it back in before it would detect the card. The solution seems to be to keep a design in flash that brings up the PCIe link so the PCIe slot will always be active on boot, then you can hot reset and/or warm reboot to load a new design.
0 Kudos
Highlighted
261 Views
Registered: ‎02-17-2020

Re: PCIe block reset for Linux PCIe development

Yes, I could do those things and I may have to resort to having some sort of PCIe design in the flash and/or the Xilinx Tandem PCIe system or just warm reboot once as I am doing now. However, the warm loading of a PCIe design with "echo "1" > /sys/pci/rescan" has worked well for me in the past, simplifies things a lot and appears to still work with Linux kernel 5.4.20 at least with older motherboard hardware. I will persevere, in the background, with trying to get that method working a bit longer :)

0 Kudos
Highlighted
Explorer
Explorer
248 Views
Registered: ‎08-14-2013

Re: PCIe block reset for Linux PCIe development

Interesting. I have never seen /etc/pci/rescan work if there wasn't previously something there. I tried that many times with no luck before writing my hot reset script. Could be highly dependent on the computer in question, the other PCIe devices present, and how linux decided to enumerate the bus, though.
0 Kudos
Highlighted
243 Views
Registered: ‎02-17-2020

Re: PCIe block reset for Linux PCIe development

Note that in order to get the "echo "1" > /sys/pci/rescan" system to work I have always, in the past, generated a local timed reset on the FPGA from design load/boot or'rd with the physical PCIe reset line. So when the new FPGA design starts running it resets the PCIe PHY and from that the rest of the board logic on its own.

0 Kudos
Highlighted
Participant
Participant
211 Views
Registered: ‎12-06-2011

Re: PCIe block reset for Linux PCIe development

I have a server-grade motherboard (Supermicro X10SAT) that actually disables a slot until the next power cycle if the system boots up and doesn't see an endpoint in that slot. This means that the only way to see an FPGA-based endpoint is to flash a bitstream into the board so that it is ready for link training by the time the BIOS starts its PCIe scan.

The above is fairly unusual behaviour for a motherboard, but you may be seeing that behaviour or some variant of it.

FPGAs probably break some of the assumptions that BIOS manufacturers make (for whom FPGAs probably aren't even on their radar). As far as they are concerned, if the BIOS scan appears to show nothing in a slot, then since the slot doesn't support hot-plugging, why keep all that slot's logic running? They might disable the slot to save some power, and that state might persist until the next warm boot (or power cycle in the case of the X10SAT).

0 Kudos