UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Visitor karakozov
Visitor
3,773 Views
Registered: ‎12-14-2016

PCIe SSD connection issue on custom Zynq board

Jump to solution

Hello to All!


I have a custom board based on Zynq XC7Z045. We connect SSD to Zynq device as stated here. I use petalinux 2016.4 and Vivado 2016.4. During project developing in Vivado  we have no errors or warning. In petalinux we set allconfiguration options related to NVME support.  After ~61 seconds from linux kernel started up SSD is disconnected with messages:

 

nvme nvme0: I/O 0 QID 0 timeout, disable controller
nvme nvme0: Identify Controller failed (-4)
nvme nvme0: Removing after probe failure status: -5

 

 

Software descripion:

 

Vivado 2016.4

Petalinux 2016.4 (+ try use different external kernels 4.0.0, 4.9.0 (from xilinx mainline github repo) with the same result. Log messages are different but SSD not available for using)

Petalinux 2017.2 - can't do petalinux-config -c kernel for unknown reason.

 

Hardware descripion:

 

SSD: Samsung 950 PRO M.2 512GB NVM Express (Model MZ - VKV512)

FPGA: XC7Z045FFG900ABX


Here my linux boot messages, and commands related to NVME and PCIe

 

[ 0.473500] xilinx-pcie 50000000.axi-pcie: PCIe Link is UP
[ 0.477591] PCI host bridge /amba_pl/axi-pcie@40000000 ranges:
[ 0.482268] No bus range found for /amba_pl/axi-pcie@40000000, using [bus 00-ff]
[ 0.488516] MEM 0x40000000..0x4fffffff -> 0x40000000
[ 0.492581] xilinx-pcie 50000000.axi-pcie: PCI host bridge to bus 0000:00
[ 0.497967] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 0.502225] pci_bus 0000:00: root bus resource [mem 0x40000000-0x4fffffff]
[ 0.507824] pci_bus 0000:00: scanning bus
[ 0.507864] pci 0000:00:00.0: [10ee:7012] type 01 class 0x060400
[ 0.507889] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
[ 0.507932] pci 0000:00:00.0: calling pci_fixup_ide_bases+0x0/0x3c
[ 0.508179] pci_bus 0000:00: fixups for bus
[ 0.508192] PCI: bus0: Fast back to back transfers disabled
[ 0.512385] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 0
[ 0.512397] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 0.519133] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 1
[ 0.519276] pci_bus 0000:01: scanning bus
[ 0.519318] pci 0000:01:00.0: [144d:a802] type 00 class 0x010802
[ 0.519360] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[ 0.519379] pci 0000:01:00.0: reg 0x18: [io 0x0000-0x00ff]
[ 0.519424] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[ 0.519457] pci 0000:01:00.0: calling pci_fixup_ide_bases+0x0/0x3c
[ 0.519762] pci_bus 0000:01: fixups for bus
[ 0.519778] PCI: bus1: Fast back to back transfers disabled
[ 0.523947] pci_bus 0000:01: bus scan returning with max=01
[ 0.523959] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 0.523971] pci_bus 0000:00: bus scan returning with max=01
[ 0.524002] pci 0000:00:00.0: BAR 0: assigned [mem 0x40000000-0x400fffff]
[ 0.529518] pci 0000:00:00.0: BAR 8: assigned [mem 0x40100000-0x401fffff]
[ 0.535022] pci 0000:00:00.0: BAR 7: no space for [io size 0x1000]
[ 0.540053] pci 0000:00:00.0: BAR 7: failed to assign [io size 0x1000]
[ 0.545408] pci 0000:01:00.0: BAR 6: assigned [mem 0x40100000-0x4010ffff pref]
[ 0.551397] pci 0000:01:00.0: BAR 0: assigned [mem 0x40110000-0x40113fff 64bit]
[ 0.557437] pci 0000:01:00.0: BAR 2: no space for [io size 0x0100]
[ 0.562455] pci 0000:01:00.0: BAR 2: failed to assign [io size 0x0100]
[ 0.567805] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 0.571544] pci 0000:00:00.0: bridge window [mem 0x40100000-0x401fffff]
[ 0.577065] pci 0000:00:00.0: fixup irq: got 0
[ 0.577074] pci 0000:00:00.0: assigning IRQ 00
[ 0.577133] pci 0000:01:00.0: fixup irq: got 168
[ 0.577142] pci 0000:01:00.0: assigning IRQ 168
[ 0.591449] xilinx-vdma 7e200000.dma: Xilinx AXI VDMA Engine Driver Probed!!
[ 0.654930] nvme nvme0: pci function 0000:01:00.0
[ 0.658281] pci 0000:00:00.0: enabling device (0140 -> 0142)
[ 0.667104] pci 0000:00:00.0: enabling bus mastering
[ 0.667141] nvme 0000:01:00.0: enabling device (0140 -> 0142)
[ 0.686386] nvme 0000:01:00.0: enabling bus mastering

 

I successfully login and after ~60 seconds from linux kernel start booting, SSD disconnecting

 

[ 61.048512] nvme nvme0: I/O 0 QID 0 timeout, disable controller
[ 61.053181] nvme nvme0: Identify Controller failed (-4)
[ 61.057023] nvme nvme0: Removing after probe failure status: -5
[ 61.061768] nvme 0000:01:00.0: PME# disabled

[ 61.083708] random: nonblocking pool is initialized

 

Here I do echo 1 > /sys/bus/pci/rescan

 

[ 167.978865] pci_bus 0000:00: scanning bus
[ 167.978903] pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 0
[ 167.978918] pci_bus 0000:01: scanning bus
[ 167.978963] pci 0000:01:00.0: [144d:a802] type 00 class 0x010802
[ 167.979011] pci 0000:01:00.0: reg 0x10: [mem 0x40110000-0x40113fff 64bit]
[ 167.979035] pci 0000:01:00.0: reg 0x18: [io 0x0000-0x00ff]
[ 167.979094] pci 0000:01:00.0: reg 0x30: [mem 0x40100000-0x4010ffff pref]
[ 167.979139] pci 0000:01:00.0: calling pci_fixup_ide_bases+0x0/0x3c
[ 167.979489] pci_bus 0000:01: bus scan returning with max=01
[ 167.979508] pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 1
[ 167.979521] pci_bus 0000:00: bus scan returning with max=01
[ 167.979552] pci 0000:00:00.0: BAR 7: no space for [io size 0x1000]
[ 167.984534] pci 0000:00:00.0: BAR 7: failed to assign [io size 0x1000]
[ 167.989905] pci 0000:01:00.0: BAR 6: assigned [mem 0x40100000-0x4010ffff pref]
[ 167.995855] pci 0000:01:00.0: BAR 0: assigned [mem 0x40110000-0x40113fff 64bit]
[ 168.001919] pci 0000:01:00.0: BAR 2: no space for [io size 0x0100]
[ 168.006878] pci 0000:01:00.0: BAR 2: failed to assign [io size 0x0100]
[ 168.012685] nvme nvme0: pci function 0000:01:00.0
[ 168.016169] nvme 0000:01:00.0: enabling bus mastering

 

after ~60 seconds from pci bus rescan, SSD disconnecting

 

[ 229.078518] nvme nvme0: I/O 0 QID 0 timeout, disable controller
[ 229.083189] nvme nvme0: Identify Controller failed (-4)
[ 229.087032] nvme nvme0: Removing after probe failure status: -5
[ 229.091698] nvme 0000:01:00.0: PME# disabled

 

lspci -xxxvvv

 

Short log:

SSD is connected

 

root@plnx_arm:~# lspci
00:00.0 PCI bridge: Xilinx Corporation Device 7012
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a802 (rev 01)

 

After ~ 60 seconds from kernel start or pci rescan


root@plnx_arm:~# lspci
00:00.0 PCI bridge: Xilinx Corporation Device 7012

 

Full log:

SSD is connected

root@plnx_arm:~# lspci -vvvxxx
00:00.0 PCI bridge: Xilinx Corporation Device 7012 (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Region 0: Memory at 40000000 (32-bit, non-prefetchable) [size=1M]
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 40100000-401fffff
Prefetchable memory behind bridge: 00000000-000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [60] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 1
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt+
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [128 v1] Vendor Specific Information: ID=0001 Rev=0 Len=038 <?>
Capabilities: [200 v1] Vendor Specific Information: ID=0002 Rev=0 Len=038 <?>
00: ee 10 12 70 46 01 10 00 00 00 04 06 10 00 01 00
10: 00 00 00 40 00 00 00 00 00 01 01 00 01 01 00 00
20: 10 40 10 40 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 00 00 01 00
40: 01 48 23 00 08 00 00 00 05 60 80 01 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 10 00 42 01 29 80 00 00 10 28 00 00 12 f4 33 00
70: 00 00 12 b0 00 00 00 00 c0 03 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a802 (rev 01) (prog-if 02 [NVM Express])
Subsystem: Samsung Electronics Co Ltd Device a801
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 168
Region 0: Memory at 40110000 (64-bit, non-prefetchable) [size=16K]
Region 2: I/O ports at <unassigned> [disabled]
Expansion ROM at 40100000 [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <4us, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [b0] MSI-X: Enable+ Count=9 Masked-
Vector table: BAR=0 offset=00003000
PBA: BAR=0 offset=00002000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [158 v1] Power Budgeting <?>
Capabilities: [168 v1] #19
Capabilities: [188 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [190 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
Kernel driver in use: nvme
00: 4d 14 02 a8 46 05 10 20 01 02 08 01 10 00 00 00
10: 04 00 11 40 00 00 00 00 01 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 4d 14 01 a8
30: 01 00 10 40 40 00 00 00 00 00 00 00 a8 01 00 00
40: 01 50 03 00 08 00 00 00 00 00 00 00 00 00 00 00
50: 05 70 86 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 b0 02 00 c0 8f 00 10 10 28 10 00 43 68 47 00
80: 00 00 12 10 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 10 08 00 00 00 00 00 00 0e 00 00 00
a0: 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 11 00 08 80 00 30 00 00 00 20 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

 

1) Please, give me any tips where it can be problem?
2) Can you tell when device file is appear /dev/nvme0n1p1? Because I have only /dev/nvme0 while SSD is connected.

 

Thank you for any help.

Below I attach all messages and config.gz in several files.

0 Kudos
1 Solution

Accepted Solutions
Highlighted
1,613 Views
Registered: ‎02-07-2008

Re: PCIe SSD connection issue on custom Zynq board

Jump to solution

@karakozov

 

I've been working on this problem for the last few days and I finally found a way to fix it.

 

Through trial and error (hence the "last few days"), I figured out that this error occurs when the BAR0 range is set to less than 1GB in the axi_pcie IP core settings (the actual range in the address map doesn't matter). Are you setting your BAR0 range to less than 1GB? The tutorial says to set BAR0 to 1GB, but this choice was entirely by chance (I wrote the tutorial). You might find that you do not have enough range in your memory map for 1GB, well that's fine, it seems that it still works if you assign less memory in the memory map (as long as it is sufficient for your end-point's BARs) and then configure the axi_pcie IP for BAR0 range of 1GB.

 

The best solution is to enable GP1 and connect the BAR0 (S_AXI) to that. This way you can assign it the entire 1GB from 0x80000000 to 0xBFFFFFFF in your memory map. The error will go away, and you don't have any mismatch between your IP settings and your address map.

 

I know this reply is probably way too late, but hopefully it helps someone else.

 

Jeff

 

 

 

View solution in original post

3 Replies
Adventurer
Adventurer
3,257 Views
Registered: ‎09-05-2007

Re: PCIe SSD connection issue on custom Zynq board

Jump to solution

same issue here with zcu102 and BPXNVME 240GB

 

did you find a solution ?

 

regards

0 Kudos
Highlighted
1,614 Views
Registered: ‎02-07-2008

Re: PCIe SSD connection issue on custom Zynq board

Jump to solution

@karakozov

 

I've been working on this problem for the last few days and I finally found a way to fix it.

 

Through trial and error (hence the "last few days"), I figured out that this error occurs when the BAR0 range is set to less than 1GB in the axi_pcie IP core settings (the actual range in the address map doesn't matter). Are you setting your BAR0 range to less than 1GB? The tutorial says to set BAR0 to 1GB, but this choice was entirely by chance (I wrote the tutorial). You might find that you do not have enough range in your memory map for 1GB, well that's fine, it seems that it still works if you assign less memory in the memory map (as long as it is sufficient for your end-point's BARs) and then configure the axi_pcie IP for BAR0 range of 1GB.

 

The best solution is to enable GP1 and connect the BAR0 (S_AXI) to that. This way you can assign it the entire 1GB from 0x80000000 to 0xBFFFFFFF in your memory map. The error will go away, and you don't have any mismatch between your IP settings and your address map.

 

I know this reply is probably way too late, but hopefully it helps someone else.

 

Jeff

 

 

 

View solution in original post

Visitor karakozov
Visitor
1,595 Views
Registered: ‎12-14-2016

Re: PCIe SSD connection issue on custom Zynq board

Jump to solution

Hi, Jeffrey!

Thank you for your answer. You are write - we have an address space less than 1GB.

Thank you for your recommendations. We will try to redesign our solution with GP1 port.

0 Kudos