The lspci and setpci commands are available natively in Linux distributions. This command has various levels of output, and provides a very useful point-in-time look at the capabilities and status of the different components trained on the PCI bus.
Most of these capabilities are reflections of the Configuration Space registers required by the PCI-Express Base Specification. As with most commands, usage instructions can be found by running "lspci --help" or "man lspci", in Linux.
By default, the lspci command will display all of the device information as shown below.
This command displays PCI devices in tree format and gives the root port BDF (Bus, Device, Function) number.
In the above log, the Xilinx device is connected to Bus Number '00', Device Number ‘01’ and Function Number ‘1’.
This is the most verbose command that displays everything. To run this command, root permission is required.
The log below shows only the portion related to the Xilinx PCIe device.
Points to note:
• Both the Link Capability Register and Link Status Register show Gen3x8. Sometimes, due to link issues, it might train down. The trained down link status is reflected in the link status register:
• The Correctable Error Status register shows Non-Fatal Error as set. During the boot, the host probes the non-configured functions as well. Because the design is configured for one function, the probe for other functions is reported as an unsupported request. This unsupported request is reported as an Advisory Non-Fatal error. If Non-Fatal Error, Unsupported Request and Correctable Error are set during the boot, this can be ignored. They can be cleared by doing a config write to the respective bits in the corresponding registers.
• Users must monitor errors in the uncorrectable error status register. If an error is reported in this register, it must be investigated and resolved.
• The PCI Express Capabilities starts at ‘80’.
lspci -vs <BDF>
This command gives verbose output for the selected device as shown below:
lspci -vvvs <BDF>
This is the same command but with more verbose output as shown below:
lspci -nvmms <BDF>
This command displays the PCI device Vendor ID and Device ID as numbers.
This command gives a hexadecimal dump of the whole PCI configuration space.
The first two bytes at 0x00 are actually 0x10EE. Within the PCIe specification, all data is defined by an offset, for example the Yellow box containing 0x80 is at offset 0x34 – which makes this the pointer to the first address of the extended capabilities register.
PCI-IDs: In the red box at 0x00 is the Vendor ID (0x10EE), followed in the blue box by the Device ID (0x7038). Down at offset 0x2C and 0x2E in the red and blue boxes respectively are the Sub-Vendor ID (0x10EE) and the Sub-Device ID (0x0700).
Bus Master Enable: Back up at 0x04 in the yellow box there is a 16 bit word “0000 0000 0000 0111”. This is the Command register. Bit 2 is "Bus Master Enable".
At 0x06 in the green box is a 16 bit word “0000 0000 0000 0001”. This is the status register, which will change over time and is a way of signalling to the root complex that certain conditions have occurred.
BAR and Memory at 0x10, there is a 32-bit word “0000 0000 0000 0000 1111 0111 1010 0000”
Bit 0 = 0 – A request for memory space
Bit 2:1 = “00” – Base address which is 32 bits wide
Bit 3 = “0” – Non-prefetchable
Bit 31:4 = 0xF7C0 (with the lowest 4 bits assumed to be 0, as these must be assigned on a byte and dword boundary) Note: if this had been a 64 bit wide request, the next D-Word would contain the upper memory address, and the follow on BAR would be BAR2.
At 0x14 – this would be the BAR1 address, however, since it is all zeros, this device only has one BAR option
Checking the PCIe Link Width
PCIe width determines the number of PCIe lanes. The command below makes it easier to find the PCIe Link Width information in the Link Capabilities Register and the negotiated link width in the Link Status Register.
Checking PCIe Speed
Similar to the command for checking the PCIe link width information, the command below provides information on PCIe speed.
Checking PCIe Max Payload Size (MPS)
The command below provides the Max Payload Size value under the Device Control Register.
Checking PCIe Max Read Request Size
Listing all PCIe Devices
The setpci command can be used for reading from and writing to configuration registers. See “setpci –help” for detailed information on setpci features.
setpci knows the names of all registers in the standard configuration headers. The “setpci –dumpregs” command shows a list of all PCI registers and capabilities as shown below:
[root@localhost xilinx]# setpci --dumpregs cap pos w name 00 W VENDOR_ID 02 W DEVICE_ID 04 W COMMAND 06 W STATUS 08 B REVISION 09 B CLASS_PROG 0a W CLASS_DEVICE 0c B CACHE_LINE_SIZE 0d B LATENCY_TIMER 0e B HEADER_TYPE 0f B BIST 10 L BASE_ADDRESS_0 14 L BASE_ADDRESS_1 18 L BASE_ADDRESS_2 1c L BASE_ADDRESS_3 20 L BASE_ADDRESS_4 24 L BASE_ADDRESS_5 28 L CARDBUS_CIS 2c L SUBSYSTEM_VENDOR_ID 2e W SUBSYSTEM_ID 30 L ROM_ADDRESS 3c B INTERRUPT_LINE 3d B INTERRUPT_PIN 3e B MIN_GNT 3f B MAX_LAT 18 B PRIMARY_BUS 19 B SECONDARY_BUS 1a B SUBORDINATE_BUS 1b B SEC_LATENCY_TIMER 1c B IO_BASE 1d B IO_LIMIT 1e W SEC_STATUS 20 W MEMORY_BASE 22 W MEMORY_LIMIT 24 W PREF_MEMORY_BASE 26 W PREF_MEMORY_LIMIT 28 L PREF_BASE_UPPER32 2c L PREF_LIMIT_UPPER32 30 W IO_BASE_UPPER16 32 W IO_LIMIT_UPPER16 38 L BRIDGE_ROM_ADDRESS 3e W BRIDGE_CONTROL 10 L CB_CARDBUS_BASE 14 W CB_CAPABILITIES 16 W CB_SEC_STATUS 18 B CB_BUS_NUMBER 19 B CB_CARDBUS_NUMBER 1a B CB_SUBORDINATE_BUS 1b B CB_CARDBUS_LATENCY 1c L CB_MEMORY_BASE_0 20 L CB_MEMORY_LIMIT_0 24 L CB_MEMORY_BASE_1 28 L CB_MEMORY_LIMIT_1 2c W CB_IO_BASE_0 2e W CB_IO_BASE_0_HI 30 W CB_IO_LIMIT_0 32 W CB_IO_LIMIT_0_HI 34 W CB_IO_BASE_1 36 W CB_IO_BASE_1_HI 38 W CB_IO_LIMIT_1 3a W CB_IO_LIMIT_1_HI 40 W CB_SUBSYSTEM_VENDOR_ID 42 W CB_SUBSYSTEM_ID 44 L CB_LEGACY_MODE_BASE 01 00 - CAP_PM 02 00 - CAP_AGP 03 00 - CAP_VPD 04 00 - CAP_SLOTID 05 00 - CAP_MSI 06 00 - CAP_CHSWP 07 00 - CAP_PCIX 08 00 - CAP_HT 09 00 - CAP_VNDR 0a 00 - CAP_DBG 0b 00 - CAP_CCRC 0c 00 - CAP_HOTPLUG 0d 00 - CAP_SSVID 0e 00 - CAP_AGP3 0f 00 - CAP_SECURE 10 00 - CAP_EXP 11 00 - CAP_MSIX 12 00 - CAP_SATA 13 00 - CAP_AF 0001 00 - ECAP_AER 0002 00 - ECAP_VC 0003 00 - ECAP_DSN 0004 00 - ECAP_PB 0005 00 - ECAP_RCLINK 0006 00 - ECAP_RCILINK 0007 00 - ECAP_RCECOLL 0008 00 - ECAP_MFVC 000a 00 - ECAP_RBCB 000b 00 - ECAP_VNDR 000d 00 - ECAP_ACS 000e 00 - ECAP_ARI 000f 00 - ECAP_ATS 0010 00 - ECAP_SRIOV
Identifying the register in setpci
Below are various ways to identify the register being used in the setpci command.
Using the Hexadecimal address
Provide the register name
For registers which are part of the PCI capability, the first register can be addressed with the name of the capability. In the --dumpregs output. Check for names starting with `CAP_' or `ECAP_'. This can be followed with +offset to add an offset (a hex number) to the address. This makes it easier to address registers which are part of the respective capability register which is set.
Width specifiers (b, .w, or .l) are used to choose how many bytes (1, 2, or 4) are to be read or written. The specifier can be dropped if the register is being referred to by its name and the width of the register is already known.
All names of registers and width specifiers are case-insensitive.
This points to the value in the command register. If this is replaced with 4.w, it will point to the same location.
This points to the value of both the command and the status register.
This points to the upper byte of the vendor ID register.
This corresponds to the second word of the power management capability.
This points to the first 32-bit word of the extended capability with ID 0x108.
setpci –s 24:00.0 04.w=6
For MSI interrupts to work, the Bus Master Enable bit must be set in the PCIe Configuration. The above command can be used to set the ‘Bus Master Enable’ bit in the command register. "24:00.0" is the BDF number in this example. It will be different for different devices and also depends on the system. For the correct BDF of the device to be addressed, see the corresponding lspci log.
A value of 6 means that we are setting the Memory Enable bit and the Bus Master Enable bit.
setpci –s 24:00.0 4a.w=1
The MSI register must also be enabled in the PCIe Configuration Space for MSI interrupts to work. In UltraScale+ devices, it is at offset 0x48 (which is also shown in the lspci log as Capabilities: ). To do this, issue a PCIe Configuration Write to set bit 16 (MSI Control register bit 0) to 1; the above command does that.
After executing the above command, run the lspci command. This should show “MSI: Enable +” instead.
setpci -s 01:00.0 82.b
The above command reads from the Link Status Register of the UltraScale+ PCIe endpoint device
The address ‘82’ is for UltraScale+ device. See (PG213) for the detailed table.
setpci -s 00:01.0 d0.b
The above command reads from the Link Control 2 Register of the Root Port
The link capability base address is a0 as shown below in the corresponding lspci log:
The Link Control 2 Register offset is '30'. Adding 'a0' to '30' results in 'd0'. The ‘a0’ address is for the root port device shown in the above screen shot. This can vary depending on the device used. For the root port using the UltraScale+ device, the Capabilities address starts at '70' as shown below and the Link Control 2 register offset will be the same ‘30’ i.e. 48 in decimal value.
The base address value of ‘70’ can also be read from the lspci log as shown below:
setpci -s 00:01.0 d0.b=42
The above command writes to the Link Control 2 Register to set the speed to Gen2.
The value ‘2’ here indicates Gen2 and the other bit is ‘Slot Clock’ which is already enabled and hence not changed. After issuing the above command, if you do lspci, it will show that the speed value in the register has changed to Gen2. However, this is just the speed that it will train to when it goes through the link training again.
To change the link speed to Gen2, the link must be retrained. This can be done by executing the command shown below:
setpci -s 00:01.0 b0.b=62
Changing The PCIe Max Read Request Size
Query the register in order to avoid overriding other properties.
setpci -s 04:00.0 78.w
Write the desired value to the register.
setpci -s 04:00.0 78.w=2936
The byte offset of the Device Control Register where the Max Read Request resides is 78h for UltraScale+ devices. This can vary for other devices. Users should consult the respective product guides. The table below is from (PG213).