UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Contributor
Contributor
32,179 Views
Registered: ‎12-02-2014

Is it possible to do enumeration without restart the PC?

Jump to solution

Hello,

 

In my PC, I have a KC705 dev board in a PCIe slot.

It works fine, but every time I re-program the FPGA via jtag, my driver lost connection with the card and I have to restart the PC to solve it.

 

As to my understanding, the Root Complex on the motherboard does the enumeration of all PCIe card at startup, once we remove a PCIe card and put it back, then the RC can no longer identify the card without a new enumeration.

 

So my question is : Is there any method to tell the RC to re-do a new enumeration? Should this be done at the firmware side or the software side?

 

 

0 Kudos
1 Solution

Accepted Solutions
Xilinx Employee
Xilinx Employee
47,149 Views
Registered: ‎11-25-2015

Re: Is it possible to do enumeration without restart the PC?

Jump to solution

Hi @colilpsc,

 

Yes my understanding is in-line with what you say. I actually note that if you just do re-scan, it only re-scan the BAR information that was already there before (so if it was 1 bar previously, then only that bar will come back up on rescan). The rescan will work correctly if remove is used first (as your 3rd test). I'm not sure what's different in the remove case, but it looks like if you do remove it, the rescan function will do full enumeration rather than just the bar that was there previously

 

What these tests didn't reveal is whether the BAR information/list is kept by the OS or BIOS after the OS loads. For example I don't know if those clear and update the BAR information at the BIOS or at the OS. I understand they do re-enumerate the BAR but which side kept that information is not clear

 

If we can understand what remove/rescan function behind the scene do then we can conclude your understanding

 

Thanks,

Sethu

View solution in original post

8 Replies
Xilinx Employee
Xilinx Employee
32,158 Views
Registered: ‎08-01-2008

Re: Is it possible to do enumeration without restart the PC?

Jump to solution
yes I think it possible check these ARs
https://www.xilinx.com/support/answers/37406.html
https://www.xilinx.com/support/answers/34777.html
Thanks and Regards
Balkrishan
--------------------------------------------------------------------------------------------
Please mark the post as an answer "Accept as solution" in case it helped resolve your query.
Give kudos in case a post in case it guided to the solution.
Highlighted
Teacher muzaffer
Teacher
32,127 Views
Registered: ‎03-31-2012

Re: Is it possible to do enumeration without restart the PC?

Jump to solution

@colilpsc take a look at this post: http://stackoverflow.com/questions/32334870/how-to-do-a-true-rescan-of-pcie-bus

you need to unload your driver on the host, do a rescan and reload your driver. This works for us really well.

- Please mark the Answer as "Accept as solution" if information provided is helpful.
Give Kudos to a post which you think is helpful and reply oriented.
Xilinx Employee
Xilinx Employee
32,090 Views
Registered: ‎11-25-2015

Re: Is it possible to do enumeration without restart the PC?

Jump to solution

Hi @colilpsc,

 

There is actually no robust solution on this.

 

For Linux, there's way you can get the bus enumeration again

 

Enumeration is done twice. Once at BIOS level and then it forwards to the OS..So if it's possible at all, the driver will need to ask the kernel/OS to re-enumerate.

 

For the linux one we have an AR for it  - http://xkb/Pages/37/37406.aspx

 

If that doesn't work there's one other way in Linux. You can do this from the terminal:

echo 1 > /sys/bus/pci/<your PCI bus number>/remove


echo 1 > /sys/bus/pci/rescan


setpci -s <your PCI bus number> 04.w=7

<your PCI bus number> is your card's bus.dev.fn number for example 02.00.0. You can get this by doing lspci and look for your card. The number is on the left hand side


The last line there, it sets Command register (offset 4) to 7 (memory/IO/bus master enable bit)

 

For Windows, we haven't found a way.

 

My Speculation was the way Windows works is it uses the enumeration value from BIOS. If that's the case, then it will never work and need to reboot

 

Thanks,

Sethu

-----------------------------------------------------------------------------------------------

Please mark the post as "Accept as solution" if the information provided answers your query/resolves your issue.

Give Kudos to a post which you think is helpful.

 

Contributor
Contributor
32,057 Views
Registered: ‎12-02-2014

Re: Is it possible to do enumeration without restart the PC?

Jump to solution

Hi @sethus,

The AR link http://xkb/Pages/37/37406.aspx, I can't open it. I think it's the same as AR# 37406 offered by @balkris?

Background:
Ubuntu 16.04, kernel version 4.4.0-47-generic
dev board: kc705
The PCIe core is configured to have 3 BARs(BAR0,1,2) of 32 bits addressable and Non-prefetchable,
BAR0 : 8 MByte
BAR1 : 1 MByte
BAR2 : 2 MByte


@muzaffer Thanks for the link.
I tested these two methods, here is what I did and the results:

First test:
step 1: Power up PC (kc705 inserted but power OFF);
step 2: Power up kc705(link led is on, user clk led blink);
step 3: use lspci, kc705 is not listed as expected;
step 4: echo 1 > /sys/bus/pci/rescan
step 5: use lspci, kc 705 is listed now;
step 6: use lspci -vvv -s <my card>, it seems that the linux rescan method can NOT detect all my BARs:

Region 1: Memory at e3300000 (32-bit, non-prefetchable) [size=1M]

There is only BAR1 correctly detected but not BAR0 and BAR2;


Second test:
step 1: power up kc705 (PC is off);
step 2: power up PC(kc705 pcie link led on, user clk led blink);
step 3: use lspci, kc705 is listed as expected;
step 4: use lspci -vvv -s <my card>, all BARs are detected correctly.
step 5: insert my driver into kernel using : "insmod"
step 6: read a register(pre-defined for test) from kc705, and it works correctly.
step 7: cp /sys/bus/pci/deviced/<my card>/config myConfig
step 8: remove my driver from kernel using "rmmod" and reset KC705 board.
step 9: use lspci -vvv -s <my card>, kc705 is still listed and other infos remains the same.
step 10: "insmod" my driver again, driver successfully loaded.
step 11: read the same register from kc705, NOT working this time, returns "FFFFFFFF";
step 12: cp myConfig /sys/bus/pci/devices/<my card>/config
step 13: re-load my driver and read that same register, it works correctly this time.


Third test:
begin after step 11 of the second test.
step 12: echo 1 > /sys/bus/pci/devices/<my card>/remove
step 13: echo 1 > /sys/bus/pci/rescan
step 14: re-load my driver and read that same register, it works correctly.


And something I learned from these tests:
BIOS won't rescan the PCI bus automatically even if there is some card insert/remove event, neither the linux.
And if linux do "rescan" pci bus, it works only if the card is enumerated when powering up the motherboard.
which means probably that the linux "rescan" feature need more or less some help from BIOS to complete those missing infomation(BAR0,2 infos in the first test)


Am I understanding it correctly?

Thanks for your answers.

colilpsc

0 Kudos
Xilinx Employee
Xilinx Employee
32,046 Views
Registered: ‎11-25-2015

Re: Is it possible to do enumeration without restart the PC?

Jump to solution

Hi Collipisc,

 

Regarding AR37406, It is intended for users who want to be able to reconfigure the FPGA but not re-boot the system, which is sometimes helpful when debugging problems. This is not intended to be used in an open system product. This is for Linux environments.

 
First issue a warm reset so that the system is up and running functionally. Then run the following command:
 
% cp /sys/bus/pci/deviced/0000:03:00:0/config myConfig
 
Where 03:00:0 is your own domain:bus:dev.func number that the BIOS chooses for you.
This will leave you with a file called myConfig with all the configuration data.
 
You can take a look at the content with the following command:
% od -x myConfig

Now the FPGA can be reconfigured using JTAG or some other method. To reinstate the PCIe configuration space in the endpoint, run the following command:
 
% cp myConfig /sys/bus/pci/devices/0000:03:00:0/config
 
The endpoint should be up and running now on your system.
 
Thanks,
Sethu
0 Kudos
Xilinx Employee
Xilinx Employee
47,150 Views
Registered: ‎11-25-2015

Re: Is it possible to do enumeration without restart the PC?

Jump to solution

Hi @colilpsc,

 

Yes my understanding is in-line with what you say. I actually note that if you just do re-scan, it only re-scan the BAR information that was already there before (so if it was 1 bar previously, then only that bar will come back up on rescan). The rescan will work correctly if remove is used first (as your 3rd test). I'm not sure what's different in the remove case, but it looks like if you do remove it, the rescan function will do full enumeration rather than just the bar that was there previously

 

What these tests didn't reveal is whether the BAR information/list is kept by the OS or BIOS after the OS loads. For example I don't know if those clear and update the BAR information at the BIOS or at the OS. I understand they do re-enumerate the BAR but which side kept that information is not clear

 

If we can understand what remove/rescan function behind the scene do then we can conclude your understanding

 

Thanks,

Sethu

View solution in original post

Explorer
Explorer
14,621 Views
Registered: ‎05-28-2014

Re: Is it possible to do enumeration without restart the PC?

Jump to solution

Hello,

 

I have exact issue, but trying the same didn't help in my case.

 

This is what I tried:

1. reboot always solves the issue

2. without reboot, I tried remove/rescan method, but it didn't help:

root@VP88x_2017_3_PRJ0191:~# echo 1 > /sys/bus/pci/devices/0000\:00\:00.0/remove
root@VP88x_2017_3_PRJ0191:~# echo 1 > /sys/bus/pci/rescan
[ 48.979275] pci 0000:00:00.0: PCI bridge to [bus 01-0c]

 Seems that the second device (01:00.0 Communication controller) is missing  !

 

In normal behaviour (when we boot without issues) it gives:

[ 74.843505] pci 0000:00:00.0: BAR 8: assigned [mem 0xe0000000-0xe00fffff]
[ 74.850239] pci 0000:01:00.0: BAR 0: assigned [mem 0xe0000000-0xe00007ff]
[ 74.856990] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
[ 74.862195] pci 0000:00:00.0: bridge window [mem 0xe0000000-0xe00fffff]
[ 74.869132] Accessing 47232 of total 1 entries in dev->bar
[ 74.874579] dev->bar[u].virt_address:cffd07c value:38011121
[ 74.880087] dev->bar[u].virt_address:cffd080 value:deadbeef
[ 74.885644] dev->bar[u].virt_address:cffd084 value:baadf00d
[ 74.891196] dev->bar[u].virt_address:cffd088 value:eba00eba
[ 74.896755] dev->bar[u].virt_address:cffd08c value:beefdeaf

 

3. I also tried the /sys/bus/pci.../config (after boot just copying the restored config, without doing anything else) but it failed as following:

root@VP88x_2017_3_PRJ0191:~# cp /mnt/pciconfig0 /sys/bus/pci/devices/0000\:00\:00.0/config
cp: can't create '/sys/bus/pci/devices/0000:00:00.0/config': File exists

 

4. I made additional try with the following command which seems to succeeded, yet the second device (01:00.0 Communication controller) is still missing  !

root@VP88x_2017_3_PRJ0191:/lib/modules# dd if=/mnt/pciconfig0 of=/sys/bus/pci/devices/0000\:00\:00.0/config
8+0 records in
8+0 records out

 

Complete log of tests and dmesg is here:

https://pastebin.com/eSi8136s

 

 

Thanks for any suggestion,

ranran

0 Kudos
Visitor xytech
Visitor
7,720 Views
Registered: ‎03-14-2019

Re: Is it possible to do enumeration without restart the PC?

Jump to solution

Hi sethus,

Quote your words : " I actually note that if you just do re-scan, it only re-scan the BAR information that was already there before (so if it was 1 bar previously, then only that bar will come back up on rescan)"

We met this same problem under Windows7 OS. could you please explain what do you mean by "re-scan"? Do you mean rigth click on Windows Device Manager and choose "Scan for hardware changes"? I guess this should work. but not sure.

However, we do achieve our goal by editoring ACPI Device Power State Register from D0(ON) to D3(OFF), and then backwards from D3 to D0. this software modification cheats the OS to think that one PCIe slot has been "hot-plugged and then inserted", and FPGA PCIe re-connects with PC successfully. Additionally, before doing above-mentioned, "PCIe hot-plug" property has to be set as "Enabled" in BIOS in advance.

Thanks.

0 Kudos