cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted
Adventurer
Adventurer
707 Views
Registered: ‎06-10-2019

XDMA doest see back to back MSI

Hi,

I am using PL XDMA root-complex on ZCU106 board. if i send back to back MSI after 1.2ms or more delay between MSI then I get call for interrupt handler function and I can see its correctly received/seen MSI but if EP issue back to back interrupts(MSI) so fast without any delay then, RC only see first interrupt and get called only one interrupt handler.

When I crosscheck with cat /proc/Interrupts and I can see only one interrupt seen, RC missed second interrupt.

Same issue seen if EP send 4 MSI back to back.

Example: if EP trigger MSI no 1 and MSI no2 back to back, Rootcomplex only see MSI 1.

Note: I am running Rootcomplex , x4, Gen3 on ZCU106 FPGA. EP device is custom chip which is perfectly works with Intel PC.

Everything works fine if i slow down back to back issuing MSI from EP but my application need to be send immediately and no point of wasting such a time.

I am sure anyone can re-produce this bug very easily.

Let me know if you need more information.

0 Kudos
14 Replies
Highlighted
Adventurer
Adventurer
642 Views
Registered: ‎06-10-2019

I just wrote simple testcase on EP to trigger MSI back to back and other end try to see how many MSI seen by Xilinx XDMA rootcomplex.

I am allocating 32 MSI and its successfully allocated, Once it allocated, I can see from cat /proc/interrupts, so its all good.

Rootcomplex only seeing one MSI, doesnt matter how many MSI you trigger back to back.

Rootcomplex only see all MSI if we add delay around 1 ms to 1.4ms as said before.

I would be really annoyed if it is Xilinx XDMA Rootcomplex IP bug. question is if MSI missed than where it goes, when I do cat /proc/interrupts to see interrupts, I see only one seen by HW and I get interrupt handler call for corresponding same interrupts.

I am using 2019.1 version.

has anyone seeing same issue?

0 Kudos
Highlighted
Adventurer
Adventurer
627 Views
Registered: ‎06-10-2019

I wrote test code to trigger msi back to back test and found 7.7uS delay must require between back to back MSI so XDMA RC IP detect MSi and call interrupt handle in kernel space code. 

I ran test in loop for 64 times with incremental delay to find out how much exact delays needed between back to back MSI so ZCU106 FPGA when XDMA PCIe code configure in Rootcomplex. I found reliable result with 7.7uS delay.

if Xilinx employee or anyone is interested than I am happy to write small kernel module and user space code for a test if that helps.

I am sure anyone can reproduce this issue easily.

I cant accept this delay 7.7us as I am shooting back to back MSI so often in my application.

Can anyone help here? is there any bug around in this area in XDMA RC IP?

 

0 Kudos
Highlighted
Adventurer
Adventurer
596 Views
Registered: ‎06-10-2019

it doesnt even work reliable with 7.7uS delay after reboot FPGA, now its need 700us ,completely unreliable. I am not sure whats happening.

Next step is to try with 16 MSI vector , presently I am asking for 32 MSI vector.

I dont know if this is going to help anything.....I am running out of idea.....

 

0 Kudos
Highlighted
Adventurer
Adventurer
591 Views
Registered: ‎06-10-2019

@liy  and @mmcnicho  and @deepeshm  do you have any thoughts on this issue?

0 Kudos
Highlighted
Adventurer
Adventurer
587 Views
Registered: ‎06-10-2019

I tried with Gen2 speed same result:( I mean setup XDMA RC with forcing Gen2 Speed Xilinx ZCU106 FPGA)

Tags (4)
0 Kudos
Highlighted
Adventurer
Adventurer
569 Views
Registered: ‎06-10-2019

FYI,

MSI capabilities register advertise0x101 (0x101 = 32 total possible vectors).

0 Kudos
Highlighted
Adventurer
Adventurer
419 Views
Registered: ‎06-10-2019

I have tried with setting up MSI vector 1, than 2, still same behavior, Looks to me RTL bug. worst thing is, I get ACK for MSI MemWr TLP so where its writing........

Now, I am waiting for reply from Xilinx XDMA experience people from this forum.

0 Kudos
Highlighted
Adventurer
Adventurer
380 Views
Registered: ‎06-10-2019

I a still waiting for response or any further direction/thought for debugging?

0 Kudos
Highlighted
Adventurer
Adventurer
320 Views
Registered: ‎06-10-2019

I am still waiting for response, as i said before, I am happy to write small user space code and kernel space code so you can re-produce issue on your board. its so easy to re-produce with simple test code.

if you do some MemWr64 to before kicking PCIe MSI this may little help but still need huge delay between two MSI, its really impossible to use with this issue.

Tags (4)
0 Kudos
Highlighted
Adventurer
Adventurer
285 Views
Registered: ‎06-10-2019

Any suggestion? I am running out of ideas.......

0 Kudos
Highlighted
Adventurer
Adventurer
239 Views
Registered: ‎06-10-2019

As per documents, there is 16 outstanding interrupts at a time if this is true, I shouldn't see this issue.

https://www.xilinx.com/support/answers/71105.html

0 Kudos
Highlighted
Adventurer
Adventurer
194 Views
Registered: ‎06-10-2019

I have changed to threaded irq handler , still doesn't help,  its really BUG in Xilinx RC RTL IP.

err = request_threaded_irq(dev_ctx->pci_dev->irq + vector, NULL, test, 0, "msitestdriver", dev_ctx);

As per my previous comments RC in MSI FIFO mode have 16 MSI depth as per documents but I think, There isn't any FIFO, I guess otherwise , i wont see this issue. presently, I am using RC in MSI Fifo mode.

PL XDMA RC IP also support MSI decode MODE, I dont know much about it,

can anyone suggest, do i need to make any changes in kernel module to make it working with MSI decode mode? does Xilinx provide any wrapper code which read and mask register to obtain MSI value for MSI decode mode?

It would be nice if someone can suggest if they have ever used MSI decode mode. do you think this would help to resolve this issue?

FYI, on downstream device connection details:  one PCIE switch and one EP to Xilinx RC.(PL XDMA RC -->PCieSwitch---->EP)

I am now using 2019.2 version.

 

0 Kudos
Highlighted
Adventurer
Adventurer
159 Views
Registered: ‎06-10-2019

I guess MSI decode mode should work. I can see code in pcie-xdma-pl.c

if (port->msi_mode == MSI_DECD_MODE) {
err = xilinx_request_misc_irq(port);
if (err)
return err;

err = xilinx_request_msi_irq(port);
if (err)
return err;

if MSI FIFO mode missing interrupt than How this is going to resolve this problem. I would like to try and see result.

0 Kudos
Highlighted
Adventurer
Adventurer
81 Views
Registered: ‎06-10-2019

I have implemented MSI decode mode and I got acceptable result, My application SW doesn't see any missing MSI so I am happy with this changes and planning to use in our project.

I am not going to accept this as solutions as I still don't know Why MSI fifo mode doesn't work if MSI FIFO vector depth is 16.

Where can I find Xilinx PCIe registers information? e.g where can i find "XILINX_PCIE_REG_RPIFR1" register details? pcie_xdma_pl.c code looks to me bit fishy on MSI fifo mode.

 

0 Kudos