UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Visitor hugo.k
Visitor
10,583 Views
Registered: ‎03-12-2015

PCIe root complex Error with PCIe to PCI adapter (mini-itx 7z045)

Hi,

 

I need to use a PCI device on the mini-itx developmenet board so I try to connect a PCIe to PCI adapter.

 

Link to PCIe to PCI adapter: http://www.startech.com/Cards-Adapters/Slot-Extension/PCI-Express-to-PCI...

I use the PCIe root complex design provided at http://zedboard.org/content/zynq-mini-itx-7z045-pcie-design-vivado-20144

 

First I try with ready_to_test example (zynq_mini_itx_7z045_pcie_design_v2014_4.zip/ready_to_test/sd_image_nic) a PCIe ethernet card is detected but if I connect PCIe to PCI adapter the kernel lock during PCI initialization.

 

After this, I try to regenrate Bitsrteam FSBL and u-boot from the hardware design. The only modification I made on the design is to make PTP_ETHERNET port external (I need Ethernet connexion). I ran synhtesis, implementation, bitstream generation and hardware export.

 

With the .hdf I create a petalinux project and I generate FSBL u-boot and kernel with petalinux tools. I have encoutered some issues with the auto generated device-tree, the Ethernet and PCIe sections was wrong so I decompile the device-tree blob provided in ready_to_test example with dtc script (script provided into kernel source repository) to get PCIe and Ethernet sections. I copy those information into the petalinux device-tree source before building (petalinux-build).

 

With this build I observe the same behavior as the ready_to_test example (a PCIe ethernet card is detected but when the adapter is connectted the kernel lock during PCI init).

 

The PCIe to PCI adapter is tested on a standard PC, it works.

 

Does anyone have try this kind of setup?

 

I attach three log of kernel PCI init.

  1. pcie_start_no_device.log: kernel log when no device connected into PCIe port (after the last line in log the kernel finish the boot and the system works).
  2. pcie_start_pcie_device.log: kernel log when a PCIe Ethernet card is connected (after the last line in log the kernel finish the boot and the system works, PCIe Ethernet card is detected via lspci).
  3. pci_start_pci_adapter.log: kernel log when the adapter is connected (after the last line in log the kernel lock, no others information are printed and no boot)
0 Kudos
7 Replies
Highlighted
Visitor dmladd
Visitor
10,321 Views
Registered: ‎04-01-2015

Re: PCIe root complex Error with PCIe to PCI adapter (mini-itx 7z045)

Sounds like the same problem I had.  Narrowed down with printk statements to drivers/pci/probe.c around line 1654.  I had inserted a printk statement at a certain line and it began working. 

 

Permanent fix was in function pci_scan_slot to add a "udelay" call

 


for (fn = next_fn(bus, dev, 0); fn > 0; fn = next_fn(bus, dev, fn)) {
     udelay(10); /* Added Empirically determined . */
     dev = pci_scan_single_device(bus, devfn + fn);
     if (dev) {
         if (!dev->is_added)
                         nr++;
         dev->multifunction = 1;
    }
}

 

 

Also note that the Xilinx provided xaxipcie.c and xaxipcie-msi.c are broken for certain manufacturers cards.  Specifically PCIe cards are only required to support 32bit access to configuraton space.  xaxipcie.c tries to use 8 and 16 bit access which doesn't work on all cards.  Also xaxipcie-msi.c is not correct either for MSI interrupts.  I submitted a patch to the Xlinix kernel, but it got deferred as they were switching to a different driver for kernel 3.18 (which also didn't work for me)

 

 

0 Kudos
Contributor
Contributor
9,723 Views
Registered: ‎12-18-2012

Re: PCIe root complex Error with PCIe to PCI adapter (mini-itx 7z045)

Hi,

I am facing an issue with my Zynq root complex design (done with vivado 2015.1) and the petalinux axi-pcie driver (I am using Petalinux 2014.4 which is based on xilinx linux kernel  3.17).

The PCIe link is up, but the configuration fails.

It seems that the BAR initialization is not correct (nothing on bus1 ?)  and I cant understand why :

----------------------------------------------------------------------------------------------------------------------------------------------------

xaxi_pcie_init_port: LINK IS UP
AXI PCIe Root Port Probe Successful
xaxi_pcie_set_bridge_resource:pci_space: 0x02000000 pci_addr:0x0000000060000000 size: 0x0000000010000000
xaxi_pcie_set_bridge_resource:Setting resource in Memory Space
PCI host bridge /amba_pl/pciex@50000000 (primary) ranges:
MEM 0x0000000060000000..0x000000006fffffff -> 0x0000000060000000
PCI: PHB MEM resource 0 = 0000000060000000-000000006fffffff [200]
PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff]
pci_bus 0000:00: root bus resource [io  0x1000-0xffff]
pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]
PCI: bus0: Fast back to back transfers disabled
PCI: bus1: Fast back to back transfers disabled
pci 0000:00:00.0: BAR 0: assigned [mem 0x60000000-0x607fffff]
pci 0000:00:00.0: BAR 8: assigned [mem 0x60800000-0x60bfffff]
pci 0000:00:00.0: PCI bridge to [bus 01]
pci 0000:00:00.0:   bridge window [mem 0x60800000-0x60bfffff]

pcieport 0000:00:00.0: enabling device (0144 -> 0146)

 

endpt: registering driver V0.1
endpt: probing PCI-Express Endpoint
endpt 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x003fffff] not claimed
endpt: pci_enable_device() failed with error -22
endpt: probe of 0000:01:00.0 failed with error -22

---------------------------------------------------------------------------------------------------------------------------------------------------

 

As your design seems to work, I am wondering what version of vivado and petalinux you used for your design.

 

Thanks.

 

0 Kudos
Xilinx Employee
Xilinx Employee
9,719 Views
Registered: ‎09-10-2008

Re: PCIe root complex Error with PCIe to PCI adapter (mini-itx 7z045)

Hi Fred,

If you are really using the mini ITX board (maybe not) then I would always get a baseline with the Avnet reference design for the board.

I've seen issues when customers altered the kernel configuration for PCIe beyond just enabling the Xilinx PCIe driver and basic PCI.

You didn't include a device tree but that would be good too. The device tree is likely OK since the driver probed ok.

The output of lspci -vv might be helpful to better understand the endpoint and the bridge configuration.

Thanks
John
0 Kudos
Contributor
Contributor
9,716 Views
Registered: ‎12-18-2012

Re: PCIe root complex Error with PCIe to PCI adapter (mini-itx 7z045)

Hi John,

Thanks for your quick answer.
Actually, I don't use the mini ITX board, but a custom board with a Zynq Z030 device.

I used the axi_pcie_v2_6 IP configured in "Root Port of PCI Express Root Complex".
Device tree has been automatically generated by the device tree generator form Petalinux 2014.4.
Here is the pcie part :
    PCIE_axi_pcie_0: pciex@50000000 {
            #address-cells = <3>;
            #size-cells = <2>;
            compatible = "xlnx,axi-pcie-1.05.a";
            interrupt-parent = <&intc>;
            interrupts = <0 57 4>;
            ranges = <0x02000000 0 0x60000000 0x60000000 0 0x10000000>;
            reg = <0x50000000 0x4000000>;
            xlnx,include-rc = <0x1>;
            xlnx,pciebar-num = <0x1>;
            xlnx,pciebar2axibar-0 = <0x00000000>;
            xlnx,pciebar2axibar-1 = <0xFFFFFFFF>;
    };

Here is my kernel config :
    Bus support  --->
        -*- PCI support                                      
        [ ] Message Signaled Interrupts (MSI and MSI-X)      
        [ ] PCI Debugging                                    
        [ ] Enable PCI resource re-allocation detection      
        < > PCI Stub driver                           
        [ ] PCI IOV support                   
        [ ] PCI PRI support                 
        [ ] PCI PASID support                   
           PCI host controller drivers  --->    
        [*] PCI Express Port Bus support                
        [ ]   Root Port Advanced Error Reporting support   
        [ ]   PCI Express ASPM control                    
        < > PCCard (PCMCIA/CardBus) support  ----   

    System Type  --->
        Xilinx Specific Options --->
            [*] Xilinx AXI PCIe host bridge support


Here is the lspci -vv output :

00:00.0 PCI bridge: Xilinx Corporation Device 7022 (prog-if 00 [Normal decode])
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Region 0: Memory at 60000000 (32-bit, non-prefetchable) [size=8M]
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00000000-00000fff
        Memory behind bridge: 00000000-000fffff
        Prefetchable memory behind bridge: 00000000-000fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data&colon; 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [60] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 1
                        ExtTag+ RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep+ BwNot+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [128 v1] Vendor Specific Information: ID=0001 Rev=0 Len=038 <?>
        Capabilities: [200 v1] Vendor Specific Information: ID=0002 Rev=0 Len=038 <?>

01:00.0 Host bridge: Racore Computer Products, Inc. Device 7022
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 89
        Region 0: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=4M]
        Capabilities: [80] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [c0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Kernel modules: endpt_driver

 

 

I will also try to change the BAR settings in my Vivado design, to see if somrthing changes with the Linux driver.

Thanks.

Fred

0 Kudos
Xilinx Employee
Xilinx Employee
9,711 Views
Registered: ‎09-10-2008

Re: PCIe root complex Error with PCIe to PCI adapter (mini-itx 7z045)

Is that endpoint a master?

Thanks
John
0 Kudos
Contributor
Contributor
9,686 Views
Registered: ‎12-18-2012

Re: PCIe root complex Error with PCIe to PCI adapter (mini-itx 7z045)

Yes, the endpoint device is a PCI Bus Master.

Fred

0 Kudos
Contributor
Contributor
9,249 Views
Registered: ‎12-18-2012

Re: PCIe root complex Error with PCIe to PCI adapter (mini-itx 7z045)

Hi,

Here is a quick update on my PCIe root port / endpoint issue.

I finally fix my issue, with some help from Xilinx.

 

The issues were :

- Linux Kernel configuration which was not correct

- The class setting in my EP device. With the one we used, the pcie subsystem of linux did not touch the BARs of the EP.

 

Here are the details :

RootPort design ( Zynq Z030 )

- fpga design done with Vivado 2015.1

- linux kernel built using Petalinux 2014.4 (kernel 3.17)

 

End Point Design ( Virtex7 XC7VX690 )

- fpga design done with Vivado 2015.2

- 256MB BAR configured, Class set to "Memory controller"

(Xilinx said to only try "Bridge device", "Memory controller", "Simple communication controller")

 

Here is the RootPort DTS pcie part :
    PCIE_axi_pcie_0: pciex@50000000 {
            #address-cells = <3>;
            #size-cells = <2>;
            compatible = "xlnx,axi-pcie-1.05.a";
            interrupt-parent = <&intc>;
            interrupts = <0 57 4>;
            ranges = <0x02000000 0 0x60000000 0x60000000 0 0x10000000>;
            reg = <0x50000000 0x10000000>;
            xlnx,include-rc = <0x1>;
            xlnx,pciebar-num = <0x1>;
            xlnx,pciebar2axibar-0 = <0x00000000>;
            xlnx,pciebar2axibar-1 = <0xFFFFFFFF>;
    };

Here is the RootPort kernel config :
    Bus support  --->
        -*- PCI support                                      
        [ *] Message Signaled Interrupts (MSI and MSI-X)      
        [ *] PCI Debugging                                    
        [ *] Enable PCI resource re-allocation detection      
        < > PCI Stub driver                           
        [ ] PCI IOV support                   
        [ ] PCI PRI support                 
        [ ] PCI PASID support                   
           PCI host controller drivers  --->    
        [ ] PCI Express Port Bus support                
        [ ]   Root Port Advanced Error Reporting support   
        [ ]   PCI Express ASPM control                    
        < > PCCard (PCMCIA/CardBus) support  ----   

    System Type  --->
        Xilinx Specific Options --->
            [*] Xilinx AXI PCIe host bridge support

 

 

Here is the RootPort startup log and lspci result:

 

root@gmb:~#
root@gmb:~# dmesg | grep pci
xaxi_pcie_init_port: LINK IS UP
xaxi_pcie_set_bridge_resource:pci_space: 0x02000000 pci_addr:0x0000000060000000 size: 0x0000000010000000
xaxi_pcie_set_bridge_resource:Setting resource in Memory Space
PCI host bridge /amba_pl/pciex@50000000 (primary) ranges:
pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff]
pci_bus 0000:00: root bus resource [io  0x1000-0xffff]
pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]
pci_bus 0000:00: scanning bus
pci 0000:00:00.0: [10ee:7022] type 01 class 0x060400
pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x3fffffff]
pci 0000:00:00.0: calling pci_fixup_ide_bases+0x0/0x3c
pci_bus 0000:00: fixups for bus
pci 0000:00:00.0: scanning [bus 01-07] behind bridge, pass 0
pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 1
pci_bus 0000:01: scanning bus
pci 0000:01:00.0: [10ee:7023] type 00 class 0x058000
pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x003fffff]
pci 0000:01:00.0: calling pci_fixup_ide_bases+0x0/0x3c
pci_bus 0000:01: fixups for bus
pci_bus 0000:01: bus scan returning with max=01
pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
pci_bus 0000:00: bus scan returning with max=01
pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 01
pci 0000:00:00.0: fixup irq: got 0
pci 0000:00:00.0: assigning IRQ 00
pci 0000:01:00.0: fixup irq: got 89
pci 0000:01:00.0: assigning IRQ 89
pci 0000:00:00.0: BAR 0: no space for [mem size 0x40000000]
pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x40000000]
pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x603fffff]
pci 0000:01:00.0: BAR 0: assigned [mem 0x60000000-0x603fffff]
pci 0000:00:00.0: PCI bridge to [bus 01]
pci 0000:00:00.0:   bridge window [mem 0x60000000-0x603fffff]
pcieport 0000:00:00.0: enabling device (0144 -> 0146)
root@gmb:~#
root@gmb:~#
root@gmb:~#
root@gmb:~# insmod /lib/modules/3.17.0-xilinx/extra/epdevice_driver.ko
epdevice: registering driver V0.2
epdevice: probing PCI-Express Endpoint
pci 0000:00:00.0: enabling bus mastering
epdevice 0000:01:00.0: enabling device (0140 -> 0142)
epdevice 0000:01:00.0: enabling bus mastering
epdevice: setup BAR0 at address 0x91800000 (4096 kB)
epdevice: ready

root@gmb:~#
root@gmb:~#
root@gmb:~#
root@gmb:~# lspci -v
00:00.0 PCI bridge: Xilinx Corporation Device 7022 (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00000000-00000fff
        Memory behind bridge: 00000000-000fffff
        Prefetchable memory behind bridge: 00000000-000fffff
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [60] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [128] Vendor Specific Information: ID=0001 Rev=0 Len=038 <?>
        Capabilities: [200] Vendor Specific Information: ID=0002 Rev=0 Len=038 <?>

01:00.0 Memory controller: Xilinx Corporation Device 7023
        Subsystem: Xilinx Corporation Device 0007
        Flags: bus master, fast devsel, latency 0, IRQ 89
        Memory at 60000000 (32-bit, non-prefetchable) [size=4M]
        Capabilities: [80] Power Management version 3
        Capabilities: [c0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: epdevice
        Kernel modules: epdevice_driver

 

So now BARs detection is done correctly, and PCIe works fine.

Hope this will help other people.

 

Fred

 

0 Kudos