UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Adventurer
Adventurer
11,705 Views
Registered: ‎09-19-2014

Device tree interrupt for PCIe peripheral with Xilinx 2015.4

Jump to solution

I am using OpenEmbedded/Yocto and Xilinx 2015.4 with my Zynq system. I was previously using build recipes from Dec 2014 and Xilinx 2014.3 before, and I got my ath9k PCIe WiFi card and hostapd (to make it a WiFi access point) working. It came time to update all the scripts to the Poky Jethro 2.0 release (from Nov 2015 - https://github.com/Xilinx/meta-xilinx/tree/jethro).
Some PCIe and WiFi things in the kernel config and device tree needed updating to make that work, so when I ran:
> iw dev wlan0 scan
it showed me the list of networks it could see. So far, so good.
However, it doesn't work as an access point with hostapd now. All the initialisation messages are fine, and by reading the output, you could only assume that it was working okay.
I did confirm that having the PCIe interrupt set incorrectly in the device tree can cause hostapd to not work at all in creating an access point, but at the same time the WiFi card can see other available networks even when the interrupt is not set correctly.
It seems that somewhere in linux-xlnx was changed to handle interrupts differently, such as in the interrupt controller system or xilinx-pcie driver, between the Dec 2014 linux-xlnx and Nov 2015 linux-xlnx code.

The device tree generated with Xilinx 2015.4 looks like:

		ZynqPS_PCIE_axi_pcie: axi-pcie@50000000 {
			#address-cells = <3>;
			#interrupt-cells = <1>;
			#size-cells = <2>;
			compatible = "xlnx,axi-pcie-host-1.00.a";
			device_type = "pci";
			interrupt-map = <0 0 0 1 &pcie_intc 1>,
					<0 0 0 2 &pcie_intc 2>,
					<0 0 0 3 &pcie_intc 3>, 
					<0 0 0 4 &pcie_intc 4>;
			interrupt-map-mask = <0 0 0 7>;
			interrupt-parent = <&intc>;
			interrupts = <0 29 4>;        <**********************************
			ranges = <0x02000000 0x00000000 0x60000000 0x60000000 0x00000000 0x10000000>;
			reg = <0x50000000 0x10000000>;  <********************************
			pcie_intc: interrupt-controller {
				#address-cells = <0>;
				#interrupt-cells = <1>;
				interrupt-controller ;
			};
		};

 

(Side note: the reg entry is generated incorrectly, and the size (second value) should be reduced from 0x10000000 to 0x1000000 (one less digit) to make it work. It complains loudly when it tries to initialise the PCIe host, and fails. Note sure why that isn't generated correctly.)

 

Anyway, I confirmed that the interrupt should be 29 (or 29 + 32 = 61) according to the hardware design in Vivado (in the old and new versions).

Firstly, this is the old WORKING version, Xilinx 2014.3. The interrupt was set to 61 (29 in device tree language).)

 

# lspci -v
00:00.0 PCI bridge: Xilinx Corporation Device 7021 (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 214
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00000000-00000fff
        Memory behind bridge: 00000000-000fffff
        Prefetchable memory behind bridge: 00000000-000fffff
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [60] Express Root Port (Slot+), MSI 00
        Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [128] Vendor Specific Information: ID=0001 Rev=0 Len=038 <?>
        Capabilities: [200] Vendor Specific Information: ID=0002 Rev=0 Len=038 <?>
        Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12

01:00.0 Network controller: Qualcomm Atheros AR93xx Wireless Network Adapter (rev 01)
        Subsystem: Qualcomm Atheros Device 3114
        Flags: bus master, fast devsel, latency 0, IRQ 61    <**************
        Memory at 60000000 (64-bit, non-prefetchable) [size=128K]
        [virtual] Expansion ROM at 60020000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/4 Maskable+ 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [300] Device Serial Number 00-00-00-00-00-00-00-00
        Kernel driver in use: ath9k
# cat /proc/interrupts
           CPU0       CPU1
 61:      10206          0       GIC  61  zynqpcie, ath9k
...
214:          0          0  PCIe-MSI   0  PCIe PME
IPI1:          0        167  Timer broadcast interrupts
IPI2:       1458       1627  Rescheduling interrupts
IPI3:          0          0  Function call interrupts
IPI4:         33         58  Single function call interrupts
IPI5:          0          0  CPU stop interrupts
IPI6:        579         43  IRQ work interrupts
IPI7:          0          0  completion interrupts
Err:          0

 

Everything works here - hostapd creates and access point.


I changed the interrupt to 28 (or 60) in the device tree, just to compare. The access point doesn't work, but the WiFi card does scan and see available networks.

# lspci -v
00:00.0 PCI bridge: Xilinx Corporation Device 7021 (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 214
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00000000-00000fff
        Memory behind bridge: 00000000-000fffff
        Prefetchable memory behind bridge: 00000000-000fffff
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [60] Express Root Port (Slot+), MSI 00
        Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [128] Vendor Specific Information: ID=0001 Rev=0 Len=038 <?>
        Capabilities: [200] Vendor Specific Information: ID=0002 Rev=0 Len=038 <?>
        Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12

01:00.0 Network controller: Qualcomm Atheros AR93xx Wireless Network Adapter (rev 01)
        Subsystem: Qualcomm Atheros Device 3114
        Flags: bus master, fast devsel, latency 0, IRQ 60    <**********************
        Memory at 60000000 (64-bit, non-prefetchable) [size=128K]
        [virtual] Expansion ROM at 60020000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/4 Maskable+ 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [300] Device Serial Number 00-00-00-00-00-00-00-00
        Kernel driver in use: ath9k
# cat /proc/interrupts
           CPU0       CPU1
 60:          0          0       GIC  60  zynqpcie, ath9k
...
214:          0          0  PCIe-MSI   0  PCIe PME
IPI1:          0       1017  Timer broadcast interrupts
IPI2:       1506       1841  Rescheduling interrupts
IPI3:          0          0  Function call interrupts
IPI4:         52         39  Single function call interrupts
IPI5:          0          0  CPU stop interrupts
IPI6:       3386         29  IRQ work interrupts
IPI7:          0          0  completion interrupts
Err:          0

Notice that the first and fourth columns have the same number - I call this "synchronised hardware/software interrupts".

 


Things look like they work on a different interrupt number model in Xilinx 2015.4 / Jethro. This is not working.

Remember, the device tree generated for 2015.4 gave me interrupt 29 (or 61).
However, here it produces a long stream of interrupt handle failure kernel messages when set that way:

[    3.333200] ->handle_irq():  c005d2bc, handle_bad_irq+0x0/0x290
[    3.339101] ->irq_data.chip(): c06d5b40, 0xc06d5b40
[    3.343962] ->action():   (null)
[    3.347174]    IRQ_NOPROBE set
[    3.350211]  IRQ_NOREQUEST set
[    3.353250] unexpected IRQ trap at vector 00
[    3.358728] irq 0, desc: ee803040, depth: 1, count: 0, unhandled: 0
[    3.364908] ->handle_irq():  c005d2bc, handle_bad_irq+0x0/0x290
...
# lspci -v
00:00.0 PCI bridge: Xilinx Corporation Device 7021 (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 172
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00000000-00000fff
        Memory behind bridge: 00000000-000fffff
        Prefetchable memory behind bridge: 00000000-000fffff
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [60] Express Root Port (Slot+), MSI 00
        Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [128] Vendor Specific Information: ID=0001 Rev=0 Len=038 <?>
        Capabilities: [200] Vendor Specific Information: ID=0002 Rev=0 Len=038 <?>
        Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12

01:00.0 Network controller: Qualcomm Atheros AR93xx Wireless Network Adapter (rev 01)
        Subsystem: Qualcomm Atheros Device 3114
        Flags: bus master, fast devsel, latency 0, IRQ 171   <******************
        Memory at 60000000 (64-bit, non-prefetchable) [size=128K]
        [virtual] Expansion ROM at 60020000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/4 Maskable+ 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [300] Device Serial Number 00-00-00-00-00-00-00-00
        Kernel driver in use: ath9k
# cat /proc/interrupts
            CPU0       CPU1
  0:         29          0      none
...
166:         29          0       GIC  61  xilinx-pcie
171:          0          0     dummy   1  ath9k
172:          0          0  Xilinx PCIe MSI   0  PCIe PME
IPI1:          0          0  Timer broadcast interrupts
IPI2:        867        742  Rescheduling interrupts
IPI3:          0          0  Function call interrupts
IPI4:         15         27  Single function call interrupts
IPI5:          0          0  CPU stop interrupts
IPI6:          0          0  IRQ work interrupts
IPI7:          0          0  completion interrupts
Err:         29

The access point didn't work, but it could still see available networks in the air.


For fun, I changed the interrupt in the device tree to 28 as before (or 60) - still Xilinx 2015.4.
It didn't have bad interrupt handler messages from the kernel now.

 

# lspci -v
EXACTLY THE SAME OUTPUT AS BEFORE
# cat /proc/interrupts
            CPU0       CPU1
166:          0          0       GIC  60  xilinx-pcie
171:          0          0     dummy   1  ath9k
172:          0          0  Xilinx PCIe MSI   0  PCIe PME
IPI1:          0          0  Timer broadcast interrupts
IPI2:        753        917  Rescheduling interrupts
IPI3:          0          0  Function call interrupts
IPI4:         36          6  Single function call interrupts
IPI5:          0          0  CPU stop interrupts
IPI6:          0          0  IRQ work interrupts
IPI7:          0          0  completion interrupts
Err:          0

Apart from the bad interrupt messages not appearing, the behaviour was the same: no access point, but it could see available networks still.

 

The interrupts look they work with a different model here. My theory is that the fourth column is the hardware interrupt value, as specified in the device tree (61 for PCIe), and the first column is the software interrupt value, which Linux kernel assigns on startup. It's as though there are separate concepts of hardware interrupt numbers, and software interrupt numbers.

Notice that the xilinx-pcie in Xilinx 2015.4 is on a separate interrupt line in the interrupt list to ath9k (which makes the device not fully work), but in Xilinx 2014.3 when it was working, they were both assigned to the same interrupt number. The dummy interrupt assigment for Xilinx 2015.4 is obviously troubling too.

 

Question: Any ideas on how to assign a proper interrupt to a PCIe peripheral, such as ath9k?


If I were to have an educated guess, I would say that there might have to be a PCIe phy entry added to the device tree. In the upgrade from 2014.3 to 2015.4, I also had to upgrade the USB host in the device tree and add a USB phy according to changes in the USB kernel driver.
https://forums.xilinx.com/t5/Embedded-Linux/Petalinux-2015-2-1-usb-not-working/m-p/654349/highlight/true#M14088
I haven't been able to read anything that confirms this yet though.

Many thanks!

 

For completeness and in case someone asks, my kernel startup messages with Xilinx 2015.4 relating to PCIe:

[    0.374263] xilinx-pcie 50000000.axi-pcie: PCIe Link is UP
[    0.374289] PCI host bridge /amba_pl/axi-pcie@50000000 ranges:
[    0.374306]   No bus range found for /amba_pl/axi-pcie@50000000, using [bus 00-ff]
[    0.374328]   MEM 0x60000000..0x6fffffff -> 0x60000000
[    0.374524] xilinx-pcie 50000000.axi-pcie: PCI host bridge to bus 0000:00
[    0.374545] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.374560] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff]
[    0.374934] PCI: bus0: Fast back to back transfers disabled
[    0.374951] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.395665] PCI: bus1: Fast back to back transfers disabled
[    0.395733] pci 0000:00:00.0: BAR 0: no space for [mem size 0x40000000]
[    0.395749] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x40000000]
[    0.395766] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    0.395785] pci 0000:01:00.0: BAR 0: assigned [mem 0x60000000-0x6001ffff 64bit]
[    0.395821] pci 0000:01:00.0: BAR 6: assigned [mem 0x60020000-0x6002ffff pref]
[    0.395835] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.395850] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    0.396006] pcieport 0000:00:00.0: enabling device (0140 -> 0142)
[    0.396244] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    0.396259] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
...
[    1.324392] ath9k 0000:01:00.0: enabling device (0140 -> 0142)
[    1.465915] ehci-pci: EHCI PCI platform driver

And my kernel config with Xilinx 2015.4 relating to PCIe:

CONFIG_GENERIC_MSI_IRQ=y
CONFIG_PCI_QUIRKS=y
CONFIG_PCI=y
CONFIG_PCI_SYSCALL=y
CONFIG_PCI_MSI=y
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_REALLOC_ENABLE_AUTO is not set
# CONFIG_PCI_STUB is not set
# CONFIG_PCI_IOV is not set
# CONFIG_PCI_PRI is not set
# CONFIG_PCI_PASID is not set
# CONFIG_PCI_HOST_GENERIC is not set
CONFIG_PCIE_XILINX=y
# CONFIG_PCI_LAYERSCAPE is not set
CONFIG_PCIEPORTBUS=y
CONFIG_PCIEAER=y
# CONFIG_PCIE_ECRC is not set
# CONFIG_PCIEAER_INJECT is not set
CONFIG_PCIEASPM=y
# CONFIG_PCIEASPM_DEBUG is not set
CONFIG_PCIEASPM_DEFAULT=y
# CONFIG_PCIEASPM_POWERSAVE is not set
# CONFIG_PCIEASPM_PERFORMANCE is not set
CONFIG_PCIE_PME=y
# CONFIG_OF_ADDRESS_PCI is not set
# CONFIG_OF_PCI is not set
# CONFIG_OF_PCI_IRQ is not set

 

0 Kudos
1 Solution

Accepted Solutions
Adventurer
Adventurer
21,740 Views
Registered: ‎09-19-2014

Re: Device tree interrupt for PCIe peripheral with Xilinx 2015.4

Jump to solution

Another update: I disabled MSI interrupts, and the ath9k wifi card works now. Perhaps it is a conflict between MSI and ath9k (which would have nothing to do with Xilinx).

 

In kernel config, I did this:

#CONFIG_GENERIC_MSI_IRQ=y
#CONFIG_PCI_MSI=y

* It no longer has handle_bad_irq kernel error messages for interrupt 61.

* Hostapd AP mode works!

* However, ath9k device is still listed as having a dummy interrupt type... oh well.

 

# cat /proc/interrupts
            CPU0       CPU1
166:       7546          0       GIC  61  xilinx-pcie
171:       7552          2     dummy   1  ath9k
IPI1:          0          0  Timer broadcast interrupts
IPI2:        804        837  Rescheduling interrupts
IPI3:          0          0  Function call interrupts
IPI4:         12         29  Single function call interrupts
IPI5:          0          0  CPU stop interrupts
IPI6:          1          0  IRQ work interrupts
IPI7:          0          0  completion interrupts
Err:          0

I'm using this as the workaround for now.

0 Kudos
2 Replies
Adventurer
Adventurer
11,520 Views
Registered: ‎09-19-2014

Re: Device tree interrupt for PCIe peripheral with Xilinx 2015.4

Jump to solution

Update: The problem could possibly be with the ath9k wifi driver instead?

 

I used an ath10k card instead, and didn't change anything (PCIe interrupt was still 29 / 61 in the device tree, and on Xilinx 2015.4) other than adding driver support in the kernel.

 

This fixes the interrupt problem!

* It no longer has the PCIe device interrupt listed as a 'dummy' type.

* It no longer has handle_bad_irq kernel error messages for interrupt 61.

* Hostapd AP mode works!

 

 

# cat /proc/interrupts
            CPU0       CPU1
166:       3273          0       GIC  61  xilinx-pcie
172:          0          0  Xilinx PCIe MSI   0  PCIe PME
173:       3273          0  Xilinx PCIe MSI   1  ath10k_pci
IPI1:          0          0  Timer broadcast interrupts
IPI2:        661        698  Rescheduling interrupts
IPI3:          0          0  Function call interrupts
IPI4:         17         24  Single function call interrupts
IPI5:          0          0  CPU stop interrupts
IPI6:          1          0  IRQ work interrupts
IPI7:          0          0  completion interrupts
Err:          0

 

# lspci -v
00:00.0 PCI bridge: Xilinx Corporation Device 7021 (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 172
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00000000-00000fff
        Memory behind bridge: 00000000-000fffff
        Prefetchable memory behind bridge: 00000000-000fffff
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [60] Express Root Port (Slot+), MSI 00
        Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [128] Vendor Specific Information: ID=0001 Rev=0 Len=038 <?>
        Capabilities: [200] Vendor Specific Information: ID=0002 Rev=0 Len=038 <?>
        Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12

01:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter
        Flags: bus master, fast devsel, latency 0, IRQ 173
        Memory at 60000000 (64-bit, non-prefetchable) [size=2M]
        [virtual] Expansion ROM at 60200000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 2
        Capabilities: [50] MSI: Enable+ Count=1/8 Maskable+ 64bit-
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 00-00-00-00-00-00-00-00
        Kernel driver in use: ath10k_pci

 

There is still a different hardware interrupt / software interrupt number model in action here, but it works.

 

I still think something in linux-xlnx at tag v2015.4.01 is broken, particularly with the ath9k driver (or the interaction between drivers). That WiFi card still doesn't work unfortunately, and I would like it to.

0 Kudos
Adventurer
Adventurer
21,741 Views
Registered: ‎09-19-2014

Re: Device tree interrupt for PCIe peripheral with Xilinx 2015.4

Jump to solution

Another update: I disabled MSI interrupts, and the ath9k wifi card works now. Perhaps it is a conflict between MSI and ath9k (which would have nothing to do with Xilinx).

 

In kernel config, I did this:

#CONFIG_GENERIC_MSI_IRQ=y
#CONFIG_PCI_MSI=y

* It no longer has handle_bad_irq kernel error messages for interrupt 61.

* Hostapd AP mode works!

* However, ath9k device is still listed as having a dummy interrupt type... oh well.

 

# cat /proc/interrupts
            CPU0       CPU1
166:       7546          0       GIC  61  xilinx-pcie
171:       7552          2     dummy   1  ath9k
IPI1:          0          0  Timer broadcast interrupts
IPI2:        804        837  Rescheduling interrupts
IPI3:          0          0  Function call interrupts
IPI4:         12         29  Single function call interrupts
IPI5:          0          0  CPU stop interrupts
IPI6:          1          0  IRQ work interrupts
IPI7:          0          0  completion interrupts
Err:          0

I'm using this as the workaround for now.

0 Kudos