MSI interrupts setup on Linux host

[arhoufiry]

We are not able to see interrupts generated by the INTC ip core.

Device used: 7A200T

Vivado tools: 2017.4

Axi memory mapped PCIe version 2.8

Linux version: Redhat 7.0

We follow the proposed solution in this thread, but the INTC core does not seem to generate an IRQ based on our single interrupt (every second).

Let me know if more details are needed, but the block diagram matches the thread below:

https://forums.xilinx.com/t5/PCI-Express/Sending-an-MSI-over-PCIe-from-axi-pcie/td-p/184674

[venkata]

The intx_msi_request input pin is positive-edge detected and synchronous to axi_aclk_out.

If your design block diagram matches with that of the post on 05-02-2016 09:09 AM in the thread - https://forums.xilinx.com/t5/PCI-Express/Sending-an-MSI-over-PCIe-from-axi-pcie/td-p/184674, then it should be fine. Please confirm.

Can you share the result when"lspci -xxxvvv" command is run on Linux?

You mention that host registered 1 MSI interrupt. Do you find INTX_MSI_Grant is asserted when this happened?

Can you probe the output signal msi_vector_width?

[arhoufiry]

Yes, we are setting the interrupt as a rising edge and it is synchronous to axi_aclk_out.

lspci -xxxvvv result:

07:00.0 Memory controller: Xilinx Corporation Device 8034
Subsystem: Xilinx Corporation Device 0007
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin ? routed to IRQ 103
Region 0: Memory at 90600000 (32-bit, non-prefetchable) [size=1M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee0f00c Data: 41e2
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 1, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Device Serial Number 00-00-00-00-00-00-00-00
Kernel driver in use: udiuDrv
00: ee 10 34 80 07 04 10 00 00 00 80 05 10 00 00 00
10: 00 00 60 90 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 ee 10 07 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 00 00 00
40: 01 48 23 00 08 00 00 00 05 60 81 00 0c f0 e0 fe
50: 00 00 00 00 e2 41 00 00 00 00 00 00 00 00 00 00
60: 10 00 02 00 29 80 64 00 10 28 00 00 11 f4 03 00
70: 00 00 11 10 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 01 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

---

The host registers 1 MSI interrupt, but none of the outputs (from the INTC and PCIe cores) are toggling, except the MSI_Enable (as you can see from the chipscope capture) that is enabled when the Linux driver is running.

[venkata]

Whether interrupt signal in the waveform is input to the interrupt controller? Can you capture the following inputs to the interrupt controller?
s_axi
s_axi_aresetn
intr
processor_rst

If possible, can you please share the bd.tcl of your design? It can help to review the GUI settings of interrupt controller and axi PCIe IP.

[stew.hansen]

We made some progress by setting the interrupt input to type edge-rising from level-rising (must be a bug).

We initialized the INTC from the microblaze, and we can get it working partially.

The current issue is getting the IRQ (INTX_MSI_Request) to be cleared based on the processor_ack (msi_ack).
We have an fsm that cycles through the processor_ack states, but the IRQ does not get cleared at state 3.

It appears we cannot get the controller to work with the fast-interrupt-mode.

Here are the register values we currently have:

ISR: 0x1

IPR: 0x1

IER: 0x1

IAR: 0x0

SIE: 0x0

CIE: 0x0

IVR: 0x0

MER: 0x3

IMR: 0x1

ILR: 0xFFFFFFFF

IVAR: 0x10

P.S. If the INTC core is packaged into a custom-IP core then you can't set the sensitivity type (default was level-high).

[pvenugo]

Hi @stew.hansen,

For fast interrupts you need to use the attribute fast_interrupt when declaring the interrupt routine.

This is necessary to tell the compiler to emit the special code required for a fast interrupt handler. 

Example code is attached.

Regards

Praveen

[stew.hansen]

Thanks @pvenugo We did get that part figured out. It ended up being the processor_ack wasn't actually being routed to the interrupt controller. After fixing the processor_ack routing it worked fine.

Also from the previous post the registers for IMR are set to one, and you can't set those unless the fast_interrupt mode is set on the FPGA.

We didn't find any attached code, where would we find that?

[pvenugo]

Code snippet attached in above post.

[venkata]

@stew.hansen

Thanks for the update. Do you have everything working fine now?

[stew.hansen]

Yes for generating a single MSI we do have it working correctly now.