Petalinux 2016.3 does not produce correct address for PCIe core

[applefat]

I have a KCU105 board which I am trying to integrate Petalinux 2016.3,  Microblaze, and a PCIE root port (e.g. AXI Bridge for PCI Epxress Gen3 Subsystem ip core). Am also using Vivado 2016.3.

I have all the necessary PCIe driver components enabled in the kernel. I am trying to get the root port to detect an attached NVMe disk; it's driver is also enabled.

Petalinux boots fine, but always says LINK IS DOWN and does not detect anything on the PCIe bus. I believe it is because Linux is reading the wrong address as I have successfully enumerated the disk on a baremetal software platform (e.g. the EXACT same bitfile, but running baremetal software instead). There is a register at offset 0x144 in the CTL0 space which provides the link status.

I don't think the petalinux tools are producing the right addresses in the DTS device tree file. Address map below.

The DTS file produced is shown below.

axi_pcie3_0: axi-pcie@20000000 {
                        #address-cells = <3>;
                        #interrupt-cells = <1>;
                        #size-cells = <2>;
                        bus-range = <0x0 0xff>;
                        compatible = "xlnx,axi-pcie-host-1.00.a";
                        device_type = "pci";
                        interrupt-map = <0 0 0 1 &pcie_intc 1>,
                                        <0 0 0 2 &pcie_intc 2>,
                                        <0 0 0 3 &pcie_intc 3>,
                                        <0 0 0 4 &pcie_intc 4>;
                        interrupt-map-mask = <0 0 0 7>;
                        interrupt-parent = <&microblaze_0_axi_intc>;
                        interrupts = <1 2>;
                        ranges = <0x02000000 0x00000000 0x0000000020000000 0x0000000020000000 0x00000000 0x10000000>;
                        reg = <0x00000000 0x200000>;
                        pcie_intc: interrupt-controller {
                                #address-cells = <0>;
                                #interrupt-cells = <1>;
                                interrupt-controller ;
                        };
                };

Note that I have tried many combinations of flipping the BAR/CTR around, changing their addr, etc, and while the output in the DTS changes after "re-getting" the hardware description file using the petalinux-config tool, they never seem to really make sense. Also note that the size in the "reg" section does not even match Vivado.

More importantly the PCIe core is marked at 20000000 even though the control reg space is at 30000000. Previously I had flipped the BAR and CTR0 with the control reg space at 10000000 and the DTS tree did put the core at 10000000. But Linux boot revealed "LINK IS DOWN" in that case as well.

Also bizarre is that REG starts at 0x0. For all other simpler DTS stanzas such as for GPIO cores, REG entry is the same as the address for that peripheral. So REG at 0x0 implies to me that Linux will literally use a base address of 0x0 to check PCIe status registers? Unless PCIe sections are treated syntactically different from other sections??

Also take a look at /proc/iomem in the boooted system. It says that PCIe starts at address 0x0--clearly this cannot be correct???

root@ct_adapter_kcu105_2:~# cat /proc/iomem
00000000-001fffff : /amba_pl/axi-pcie@20000000
44a01000-44a0ffff : serial
44a10000-44a1ffff : /amba_pl/axi_quad_spi@44a10000

In short, cannot make heads nor tails of the PCIe entry in the DTS file and therefore assume something must be translating incorrectly....

[applefat]

Since I cannot edit my original post. Let me also add that after booting, I can query the actual physical address where I KNOW the status register should be, based on my Vivado hardware. Sure enough it contains the code which indicates LINK UP.

root@ct_adapter_kcu105_2:~# peek 0x30000144
0x00000882

Not sure how to convince the petatlinux tools as to the actual hardware config....

[jeffrey.johnson]

Page 4 of the datasheet for AXI Bridge for PCIe Gen3 Subsystem says "Supported S/W Driver: Not provided".

Are you sure that the AXI Bridge for PCIe Gen3 is even supported by PetaLinux?

[applefat]

Jeffrey, 

I thought I'd post my reply to your PM here since it's just general info which might help somebody else.

The baremetal enumeration testbench that I referred to is a Xilinx example from the SDK. Path should be something like this; just copy it into a new project along with any dependencies you find. It will at least give you "link up/down" and list the devices.

\Xilinx\SDK\2016.3\data\embeddedsw\XilinxProcessorIPLib\drivers\axipcie_v3_0\examples\xaxipcie_rc_enumerate_example.c

I don't think there is any problem seeing the "generic" pcie driver in SDK.

In the end I sort of gave up on the system I had constructed and used one I found elsewhere. The files to produce a system which worked for me nearly "out of the box" is here. I say nearly because I only had to change the pin map for the PCIe lanes and the refclk to get it going. Once everything booted I could see the disk.

https://github.com/fpgadeveloper/fpga-drive-aximm-pcie/tree/master/Vivado

The associated article is here:

http://www.fpgadeveloper.com/2016/04/connecting-an-ssd-to-an-fpga-running-petalinux.html

I notice it is written by "Jeff Johnson", is that not you?

I compared my own, and their design exhaustively but could not find substantial differences. So in the end I'm not entirely sure what the main problem was. But I did learn some things.

--there was some changes to the PCIe reset logic in the 2016.3 PCIe core such that there are two separate reset outputs. One just follows the input reset and the other will always be asserted when there is no link. That means if you attach it to your AXI crossbar in front of the IP (as recommended in the datasheet) , the whole Microblaze system will lock up when it tries to access that memory mapped location when there is no PCIe device attached! Extremely goofy design. The lesson is, be 100% sure your bus reset signals are not being asserted when Microblaze tries to read/write through them otherwise everything stalls (since MB has no timeout feature, unlike Zynq).

--when using petalinux-boot I found I have to hard-reset my board after every run of petalinux-boot, otherwise subsequent attempts ALWAYS fail (an invisible kernel panic, as described in the original post).  Soft-reset is not good enough.

There's no specific detail provided on the inner workings of petalinux-boot, beyond what is printed on the console in verbose mode, so  nobody can say for sure. But  since it pushes the executable files into DRAM, there must be some proxy code that it pushes over onto the processor to facilitate image transfer from JTAG to DRAM. After transferring (with "dow"), the instruction pointer is set to the start of DRAM. But somehow on subsequent runs, this command is either ignored, or some garbage instruction left over from the proxy loader is executed, or something else--either way it doesn't behave properly unless RAM and everything else is cleared out. Another undocumented problem which people have to discover on their own.

[applefat]

Forgot to add: even if there is a PCIe device attached, if the PCIe refclk is not running for whatever reason, Petalinux will also lock up on boot. The instant PetaLinux accesses hardware which is unclocked or held in reset, it will lock up. Even using the JTAG debugger there is no real way to detect this situation since JTAG debug commands just randomly start to fail as well.