cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
AntonRG
Participant
Participant
1,106 Views
Registered: ‎02-13-2019

nvme timeout, completion polled Vivado 2021.1, Petalinux 2021.01

Jump to solution

I am getting an issue with a PCIe NVMe when running petalinux 2021.01.

To try and rule out configuration issues in my project I went back to just using the Xilinx supplied BSP for the ZCU102 in Vivado 2021.01 and Petalinux 2021.01 and not using my own XSA.

When I enable nvme in the kernel config and then boot up I get the following messages and the boot process takes about 5 to 10 mins; any accessing of the drive also takes mins to complete.

[    7.190723] hid-generic 0003:046D:C077.0003: input: USB HID v1.11 Mouse [Logitech USB Optical Mouse] on usb-xhci-hcd.1.auto-1.4/input0
[   64.591575] nvme nvme0: I/O 4 QID 0 timeout, completion polled
[  101.437996] random: crng init done
[  126.031565] nvme nvme0: I/O 5 QID 0 timeout, completion polled
[  187.471563] nvme nvme0: I/O 6 QID 0 timeout, completion polled
[  248.911566] nvme nvme0: I/O 7 QID 0 timeout, completion polled
[  310.351565] nvme nvme0: I/O 4 QID 0 timeout, completion polled
[  371.791569] nvme nvme0: I/O 5 QID 0 timeout, completion polled
[  433.231568] nvme nvme0: I/O 6 QID 0 timeout, completion polled
[  494.671566] nvme nvme0: I/O 7 QID 0 timeout, completion polled
[  556.111570] nvme nvme0: I/O 4 QID 0 timeout, completion polled
[  617.551572] nvme nvme0: I/O 5 QID 0 timeout, completion polled
[  678.991571] nvme nvme0: I/O 6 QID 0 timeout, completion polled
[  740.431573] nvme nvme0: I/O 7 QID 0 timeout, completion polled
[  801.871573] nvme nvme0: I/O 4 QID 0 timeout, completion polled
[  801.877438] nvme nvme0: 4/0/0 default/read/poll queues
[  863.311582] nvme nvme0: I/O 5 QID 0 timeout, completion polled
[  924.751577] nvme nvme0: I/O 6 QID 0 timeout, completion polled
[  986.191576] nvme nvme0: I/O 7 QID 0 timeout, completion polled
[ 1047.631578] nvme nvme0: I/O 4 QID 0 timeout, completion polled
[ 1078.351604] nvme nvme0: I/O 64 QID 2 timeout, aborting
[ 1078.356764]  nvme0n1:

 

I also get the following after leaving it for longer:

[ 1109.071615] nvme nvme0: I/O 72 QID 2 timeout, aborting
[ 1139.418958] udevd[269]: worker [276] /devices/platform/axi/fd0e0000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/nvme/nvme0/nvme0n1 is taking a long time
[ 1139.791588] nvme nvme0: I/O 5 QID 0 timeout, completion polled
[ 1139.797425] nvme nvme0: I/O 256 QID 3 timeout, completion polled
[ 1139.803438] nvme nvme0: Abort status: 0x0
[ 1139.807440] nvme nvme0: Abort status: 0x0
[ 1170.511592] nvme nvme0: I/O 257 QID 3 timeout, aborting

 

 

I see that the answer record: https://www.xilinx.com/support/answers/71105.html seems related but that is for an earlier version of Vivado 2018.1 and its for the DMA/Bridge IP and not the built in PCIe of the MPSOC of the Ultrascale+ of the ZCU102.

So while I am sure its related I am not sure what the correct fix should be.

If anyone could help that would be very much appreciated.

0 Kudos
1 Solution

Accepted Solutions
matta
Visitor
Visitor
815 Views
Registered: ‎11-20-2020

I got a response from Xilinx outside of this thread, and I was told the following:

> I found out some information related to device compatibility.

PS-PCIe does not support MSI-X interrupts. If the NVMe device requires/requests MSI-X, I believe this will cause an incompatibility.

 

The PS instantiation will only work with drives not specifically limited to MSI or MSI-X interrupts.  Therefore, drive evaluation will have to be done for compatibility.  So far, I have the two following working fine:

Samsung 980 PRO

Crucial P5

View solution in original post

10 Replies
pvenugo
Moderator
Moderator
1,082 Views
Registered: ‎07-31-2012

@AntonRG , 

Please clarify if you are using PS PCIe or PL PCIe IP?

If its PS PCIe here is our AR published https://www.xilinx.com/support/answers/76169.html . If it PL let me know the IP configuration details.

 

Please share the complete boot log.

Regards

Praveen


-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
AntonRG
Participant
Participant
1,080 Views
Registered: ‎02-13-2019

Hi @pvenugo 

Thanks for the response.

I can confirm it is the PS side PCIe.

I will follow through with the AR you posted and not use the ZCU102 BSP. If that fixes the issue ill come back and accept it as the solution.

Thanks again for your time.

 

0 Kudos
AntonRG
Participant
Participant
1,055 Views
Registered: ‎02-13-2019

Hi @pvenugo ,

 

I have gone through the document and the issue still persists with this M2. I can confirm the card and M2 work fine on a standard AMD64 machine.

As a side note there is a mistake in the document on page 4 on this line:

"petalinux-create --type project --template zynqMP –name < project_name > "
-name is listed rather than --name, a small thing but I thought you might want to know.

So at present its running but it takes 15 mins to boot while it slowly spits out:

 

[   64.591516] nvme nvme0: I/O 8 QID 0 timeout, completion polled
[  126.031506] nvme nvme0: I/O 9 QID 0 timeout, completion polled
[  187.471506] nvme nvme0: I/O 10 QID 0 timeout, completion polled
[  248.911507] nvme nvme0: I/O 11 QID 0 timeout, completion polled
[  310.351508] nvme nvme0: I/O 8 QID 0 timeout, completion polled

 

The PCIe card is the SupaGeek M.2 SSD to PCIe Express 3.0 x4 Adapter Card, and the M2 is the WD_BLACK SN850

 

0 Kudos
Fernandez5
Newbie
Newbie
1,005 Views
Registered: ‎07-08-2021

If more than this are received from downstream devices, the Interrupt FIFO will overflow.

0 Kudos
matta
Visitor
Visitor
953 Views
Registered: ‎11-20-2020

I am having the same issue with certain NVMe drives.  Some work, while others produce this exact error.  When you say:

If more than this are received from downstream devices, the Interrupt FIFO will overflow.

What do you mean by "If more than this are received"?  Is there a way to clear up this error so all NVMe drives will work properly?  I have looked at all the Kernel driver options, and I definitly have it setup correctly.  Is there something that needs to be done to the Zynq in Vivado that I am not accounting for when exporting the .xsa?

0 Kudos
AntonRG
Participant
Participant
943 Views
Registered: ‎02-13-2019

Glad im not the only one

Yeah this SN850 is a very high-end M2.
I have tried it on a low-end Samsung drive and it works fine.

I know I've had issues before on other devices for things like the Raspberry Pi PCIe where you can run out of BAR space when using some cards especially the faster higher-end versions. Could something like that be the case here?

0 Kudos
matta
Visitor
Visitor
816 Views
Registered: ‎11-20-2020

I got a response from Xilinx outside of this thread, and I was told the following:

> I found out some information related to device compatibility.

PS-PCIe does not support MSI-X interrupts. If the NVMe device requires/requests MSI-X, I believe this will cause an incompatibility.

 

The PS instantiation will only work with drives not specifically limited to MSI or MSI-X interrupts.  Therefore, drive evaluation will have to be done for compatibility.  So far, I have the two following working fine:

Samsung 980 PRO

Crucial P5

View solution in original post

AntonRG
Participant
Participant
721 Views
Registered: ‎02-13-2019

Cheers for the info.

That indeed does appear to be the case.

It's a shame but it is what it is I guess.

0 Kudos
matta
Visitor
Visitor
697 Views
Registered: ‎11-20-2020

I will add that if MSI/MSI-X interrupts are required, a PL instantiation of the PCIe root complex or endpoint will be required.  This will also give the benefit of full PCIe Gen 3.

0 Kudos
AntonRG
Participant
Participant
678 Views
Registered: ‎02-13-2019

Yeah, that's a route we have gone down before also using the FPGA Drive FMC card as a starting point.
Thanks for your help.

0 Kudos