AC701 PCIe Example Design Problems

[bhall0107]

Hello,

I recently received a AC701 development board and have been trying to get any PCIe example design to work on Windows 10. Here's what I have tried:

First, I created an example design of the 7 Series PCIe Integrated Block with the default settings and X4 selected. In the XDC, I added the constrains below.

I succesfully generated the bitstream, MCS, and programmed the SPI flash memory. After I shut down the PC and power back on, the board is shown in Device Manager under Other Devices as PCI Memory Controller. What do I do from here? Should I just see if it is working by using XTP227?

I would prefer to use the DMA/Bridge for PCIe (XDMA) IP, as it already has a DMA built in and there are drivers available. Every time I create an example project using similar settings to what I used in the example above, nothing is shown in Device Manager. I used the XDC file below. 

Because of this, I can't attempt to follow Xilinx Answer 65444. I used ILA to see what was happening and noticed that the axi_aresetn (out of the XDMA core) is always asserted. The core seems dead, and there is no clock coming out of axi_aclk either. It seems like I am not constraining some IO correctly or something like that. Does anyone have an example design for the XDMA IP that works with the AC701? Or at least, does anyone have advice on how to move forward from here?

Thanks in advance for your help.

Brad

[csattar]

hi @bhall0107 ,

can you let us know the following information 

1. which version of vivado your using.

2. what are steps you have followed to create the design 

3. i believe and assume that PCIe DMA does not have default drivers , we need to install drivers frist ,

4. were able to see , Displays a watermark with the text "Test Mode" in the lower-left corner of the desktop to remind users that the system has test-signing enabled.

4. Is the drivers installed first ,insert the FPGA card and restart .

chandra 

5. 

[bhall0107]

Hello,

I'm using Vivado 2019.2. I don't see anything related to "Test Mode." Is that something I would see in Vivado or my Windows desktop?

My steps are basically explained in the first post. All I did was click "Open IP Example Design..." on the 7 Series PCIe Integrated Block and XDMA IPs. Once the example project was opened, I entered the constraints that I thought were correct for the AC701 development board. I generated the bitstream and programmed a MCS file into the board's flash memory.

I did not install the XDMA driver yet, because Xilinx Answer 65444 says to do it once I see that the device is shown as "PCI Memory Controller". I have only seen that come up for the 7 Series PCIe Integrated Block and not the XDMA IP. Should I try to install it before even plugging the card in? Do you know of any Windows settings that I would need to change?

Please let me know if you need any more information from me.

[aforencich]

The difference in the IBUFDS name is making me suspicious - IBUFDS_GTE2_X0Y3 vs IBUFDS_GTE2_X0Y2. Make sure the PCIe pins used are the same; you may need to do a bit of digging - possibly including switching from 'basic' to 'advanced' mode - to select the proper PCIe hard core site and transceiver sites. If the hard IP core example design works, look at the pinout log file to see what pins were used on that design as well as the implemented design to see which PCIe hard IP site was used, then make sure the other design is using the same pins and hard IP site. My guess is that it isn't, and as a result the transceivers on the FPGA aren't connected to the host and the link isn't coming up. It may be as simple as selecting the correct PCIe hard IP core instance, but you may need to change another setting or two as well.

[bhall0107]

@aforencich That's a good catch, thanks. It's strange, I thought that I had pasted the same IBUFDS constraint into the XDC, so maybe Vivado changed it automatically or something. Not sure if that's even possible, but I don't remember changing it. I'll take your advice and look into the hard IP site, transceiver, and pin placement. I'll let you know what I find. 

Thanks,

Brad

[bhall0107]

@aforencich @csattar , Here's what I found...

• There is only one PCIe hard core in the Artix-7 that we are using, so it is definitely being placed in the same location.
• The IO pin (sys_clk_p, sys_clk_n, sys_rst_n) constraints are being placed in the same location (E11,F11,M20).
• refclk_ibuf is getting placed into IBUFDS_GTE2_X0Y2 for both designs even though Vivado automatically added the constraint to IBUFDS_GTE2_X0Y3 in the XDC for the 7 Series PCIe Integrated Block example.

The only differences in placement that I saw were that the MMCMs are placed in different locations along with various buffers that are created by the example designs. I assume this is part of the problem. Maybe the design doesn't satisfy the requirements in PG054 pages 187-188 and 197-200?

I don't have time to dig deeper into this. Example designs should work with minimal effort especially when targeting a Xilinx evaluation board. In IP Integrator, why is there no PCIe option in the board tab like there is for different boards (i.e. Kintex KCU 105, shown below)?

Does anyone know if there an easier way to implement PCIe with DMAs on the AC701? Maybe by using the 7 Series PCIe Integrated Block and connecting DMAs to that? I would need test or example drivers that we could develop for Linux and Windows. Are there other FPGAs (or evaluation kits) where this is easier to achieve?

I'd really appreciate any advice or assistance. Thanks!

[bethe]

Hi @bhall0107 

I would recommend building in the following:

- Select AC701

- IP Catalog - Selection DMA/Bridge IP

- Configure to your settings

- Right Click, Open example design

- Build

You shouldn't need to change any of the constraints.

If that is not showing up in device manager, can you give the machine a quick *warm* reboot?  The DMA core takes significantly longer to load into the FPGA, and in some hosts it is too slow, and misses enumeration.  On a cold reboot, the FPGA loses power and has to reload, but on a warm reboot, the system just re-enumerates.

If that doesn't work, if you could add a JTAG Debugger (on the Debug tab in core configuration, prior to open example design -- will need to open new example design) and see where the LTSSM is getting stuck?

[bhall0107]

Hi @bethe, thanks for your response.

I followed the steps you listed "- Select AC701, IP Catalog - Selection DMA/Bridge IP, - Configure to your settings, - Right Click, Open example design, - Build" without making any manual edits to the XDC file. Here's what happened:

This is why I added constraints as described in my original post. 

I have tried warm reboots on XDMA example designs with constraints added, and nothing seems to change. That suggestion makes sense, and I'm not sure why it doesn't help. Do you have any ideas why it wouldn't? Are my constraints wrong? Maybe it has to do with the other axi_resetn assertion conditions (from PG054 page 185) : 

• PLL within the Core Wrapper: Loses lock, indicating an issue with the stability of the
  clock input.
• Loss of Transceiver PLL Lock: Any transceiver loses lock, indicating an issue with the
  PCI Express Link.

I do not see the Debug tab or the option for a JTAG Debugger on the XDMA core configuration (see below). I've seen it when targeting other boards, but not the AC701.

Lastly, do I need to select anything in the Shared Logic tab? This post mentions something about it, but it's a little unclear... https://forums.xilinx.com/t5/PCIe-and-CPM/DRC-Failure-on-PCIe-IP-Implementation/td-p/918375 

Thanks for your time and support.
Brad

[bethe]

Hi @bhall0107 

After writing the post, I was running a new example just to be sure.  It does look like there are some issues in there, so I am having it run in hardware today, and will post as soon as we figure out what is going on.  For sure, we need to get the XDC issues fixed on the board files. 

[bhall0107]

@bethe Thanks for looking into it. One thing I found was increasing the bitstream configuration frequency (BITSTREAM.CONFIG.CONFIGRATE) to 33MHz allows the FPGA to be configured before the host asserts the PERSTN signal. Now XDMA asserts the link up signal. The problem is that the host does not boot. It seems to be stuck before/on BIOS. Have you seen anything like this before? 
Hope that information is helpful.

Thanks again.

[bhall0107]

Hi @bethe,

Have you made any progress on this? We've been exploring other IP options, but would like to use XDMA on the Artix.

Thanks.

Brad

[nbradley777]

I too am interested in what you discover. I am having issues with getting XDMA to work for a Vivado project targeted at the AC701 board. I have ran into all the same problems as OP, using Vivado 2019.2.

Thanks.

[bethe · Accepted solution]

Hi all,

Can you add these constraints to the top level:

###############################################################################
# Pinout and Related I/O Constraints
###############################################################################

#
# SYS reset (input) signal. The sys_reset_n signal should be
# obtained from the PCI Express interface if possible. For
# slot based form factors, a system reset signal is usually
# present on the connector. For cable based form factors, a
# system reset signal may not be available. In this case, the
# system reset signal must be generated locally by some form of
# supervisory circuit. You may change the IOSTANDARD and LOC
# to suit your requirements and VCCO voltage banking rules.
# Some 7 series devices do not have 3.3 V I/Os available.
# Therefore the appropriate level shift is required to operate
# with these devices that contain only 1.8 V banks.
#

set_property IOSTANDARD LVCMOS33 [get_ports sys_rst_n]
set_property LOC M20 [get_ports sys_rst_n]
set_property PULLUP true [get_ports sys_rst_n]

###############################################################################
# Physical Constraints
###############################################################################
#
# SYS clock 100 MHz (input) signal. The sys_clk_p and sys_clk_n
# signals are the PCI Express reference clock. Virtex-7 GT
# Transceiver architecture requires the use of a dedicated clock
# resources (FPGA input pins) associated with each GT Transceiver.
# To use these pins an IBUFDS primitive (refclk_ibuf) is
# instantiated in user's design.
# Please refer to the Virtex-7 GT Transceiver User Guide
# (UG) for guidelines regarding clock resource selection.
#

#set_property LOC IBUFDS_GTE2_X0Y3 [get_cells refclk_ibuf]
set_property LOC F11 [get_ports sys_clk_p]
###############################################################################
# Timing Constraints
###############################################################################
#
create_clock -name sys_clk -period 10 [get_ports sys_clk_p]
#
#
set_false_path -to [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]
set_false_path -to [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S1}]
#
#
set_case_analysis 1 [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]
set_case_analysis 0 [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S1}]
set_property DONT_TOUCH true [get_cells -of [get_nets -of [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]]]

set_clock_group -name async1 -asynchronous -group [get_clocks {sys_clk}] -group [get_clocks {clk_125mhz}]
set_clock_group -name async2 -asynchronous -group [get_clocks {sys_clk}] -group [get_clocks {clk_250mhz}]
#
#
# Timing ignoring the below pins to avoid CDC analysis, but care has been taken in RTL to sync properly to other clock domain.
#
#
##############################################################################
# Tandem Configuration Constraints
###############################################################################

set_false_path -from [get_ports sys_rst_n]

[bhall0107]

Thanks @bethe, we were able to get the XDMA example design working on the AC701 board using these constraints. There were a few things that I wanted to mention though...

First, the example design did not work with Shared Clocking enabled. The host PC would not boot. The design did not meet timing and gave the following critical warnings:

Next, the example design worked with Shared Clocking disabled. I was able to boot the machine and use the example programs to verify that it was operational. The following critical warnings still appeared:

I commented out all of the constraints that caused critical warnings in implementation and everything still worked fine. It looks like they were meant for the 7 Series PCIe Integrated Block example design. Is that true? Should I edit them so they apply to the XDMA example design?

I am wondering which constraints you added actually made a difference. It seems like the set_clock_group constraints must have done something. I also noticed that set_property LOC IBUFDS_GTE2_X0Y3 was commented out. Was that for a specific reason? I see that the buffer was placed at site IBUFDS_GTE2_X0Y2.

Thanks again for your time and support on getting this to work. We're glad that we can move forward using XDMA.

Brad

[likewise]

Although I think you are up and running, you mention you have seen the link come up but the (x86?) PC hangs during booting.

I have seen this issue occur when the PCIe device class configured in XDMA (or PCIe block for that matter) is a Serial Controller (UART), and a x86 PC BIOS attempts to interact with it (which fails, because it does not fulfill that class).

Changing the class (to a memory controller) solves this issue.