cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
bhall0107
Adventurer
Adventurer
2,295 Views
Registered: ‎11-13-2018

AC701 PCIe Example Design Problems

Jump to solution

Hello,

I recently received a AC701 development board and have been trying to get any PCIe example design to work on Windows 10. Here's what I have tried:

First, I created an example design of the 7 Series PCIe Integrated Block with the default settings and X4 selected. In the XDC, I added the constrains below.

pcie_xdc.png

I succesfully generated the bitstream, MCS, and programmed the SPI flash memory. After I shut down the PC and power back on, the board is shown in Device Manager under Other Devices as PCI Memory Controller. What do I do from here? Should I just see if it is working by using XTP227?



I would prefer to use the DMA/Bridge for PCIe (XDMA) IP, as it already has a DMA built in and there are drivers available. Every time I create an example project using similar settings to what I used in the example above, nothing is shown in Device Manager. I used the XDC file below. 

 

dma_xdc.png

Because of this, I can't attempt to follow Xilinx Answer 65444. I used ILA to see what was happening and noticed that the axi_aresetn (out of the XDMA core) is always asserted. The core seems dead, and there is no clock coming out of axi_aclk either. It seems like I am not constraining some IO correctly or something like that. Does anyone have an example design for the XDMA IP that works with the AC701? Or at least, does anyone have advice on how to move forward from here?

Thanks in advance for your help.

Brad

0 Kudos
Reply
1 Solution

Accepted Solutions
bethe
Xilinx Employee
Xilinx Employee
1,794 Views
Registered: ‎12-10-2013

Hi all,

 

Can you add these constraints to the top level:

###############################################################################
# Pinout and Related I/O Constraints
###############################################################################

#
# SYS reset (input) signal. The sys_reset_n signal should be
# obtained from the PCI Express interface if possible. For
# slot based form factors, a system reset signal is usually
# present on the connector. For cable based form factors, a
# system reset signal may not be available. In this case, the
# system reset signal must be generated locally by some form of
# supervisory circuit. You may change the IOSTANDARD and LOC
# to suit your requirements and VCCO voltage banking rules.
# Some 7 series devices do not have 3.3 V I/Os available.
# Therefore the appropriate level shift is required to operate
# with these devices that contain only 1.8 V banks.
#

set_property IOSTANDARD LVCMOS33 [get_ports sys_rst_n]
set_property LOC M20 [get_ports sys_rst_n]
set_property PULLUP true [get_ports sys_rst_n]

###############################################################################
# Physical Constraints
###############################################################################
#
# SYS clock 100 MHz (input) signal. The sys_clk_p and sys_clk_n
# signals are the PCI Express reference clock. Virtex-7 GT
# Transceiver architecture requires the use of a dedicated clock
# resources (FPGA input pins) associated with each GT Transceiver.
# To use these pins an IBUFDS primitive (refclk_ibuf) is
# instantiated in user's design.
# Please refer to the Virtex-7 GT Transceiver User Guide
# (UG) for guidelines regarding clock resource selection.
#

#set_property LOC IBUFDS_GTE2_X0Y3 [get_cells refclk_ibuf]
set_property LOC F11 [get_ports sys_clk_p]
###############################################################################
# Timing Constraints
###############################################################################
#
create_clock -name sys_clk -period 10 [get_ports sys_clk_p]
#
#
set_false_path -to [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]
set_false_path -to [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S1}]
#
#
set_case_analysis 1 [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]
set_case_analysis 0 [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S1}]
set_property DONT_TOUCH true [get_cells -of [get_nets -of [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]]]

set_clock_group -name async1 -asynchronous -group [get_clocks {sys_clk}] -group [get_clocks {clk_125mhz}]
set_clock_group -name async2 -asynchronous -group [get_clocks {sys_clk}] -group [get_clocks {clk_250mhz}]
#
#
# Timing ignoring the below pins to avoid CDC analysis, but care has been taken in RTL to sync properly to other clock domain.
#
#
##############################################################################
# Tandem Configuration Constraints
###############################################################################

set_false_path -from [get_ports sys_rst_n]

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------

View solution in original post

14 Replies
csattar
Moderator
Moderator
2,277 Views
Registered: ‎05-02-2017

 

hi @bhall0107 ,

 

can you let us know the following information 

 

1. which version of vivado your using.

2. what are steps you have followed to create the design 

3. i believe and assume that PCIe DMA does not have default drivers , we need to install drivers frist ,

4. were able to see , Displays a watermark with the text "Test Mode" in the lower-left corner of the desktop to remind users that the system has test-signing enabled.

4. Is the drivers installed first ,insert the FPGA card and restart .

 

chandra 

 

5. 

Regards
Chandra sekhar
----------------------------------------------------------------------------------------------
Kindly note- Please mark the Answer as "Accept as solution" if solution provided is helpful.

Give Kudos to a post which you think is helpful and reply oriented.
----------------------------------------------------------------------------------------------
0 Kudos
Reply
bhall0107
Adventurer
Adventurer
2,253 Views
Registered: ‎11-13-2018

Hello,

I'm using Vivado 2019.2. I don't see anything related to "Test Mode." Is that something I would see in Vivado or my Windows desktop?

My steps are basically explained in the first post. All I did was click "Open IP Example Design..." on the 7 Series PCIe Integrated Block and XDMA IPs. Once the example project was opened, I entered the constraints that I thought were correct for the AC701 development board. I generated the bitstream and programmed a MCS file into the board's flash memory.

I did not install the XDMA driver yet, because Xilinx Answer 65444 says to do it once I see that the device is shown as "PCI Memory Controller". I have only seen that come up for the 7 Series PCIe Integrated Block and not the XDMA IP. Should I try to install it before even plugging the card in? Do you know of any Windows settings that I would need to change?

Please let me know if you need any more information from me.

0 Kudos
Reply
aforencich
Explorer
Explorer
2,224 Views
Registered: ‎08-14-2013
The difference in the IBUFDS name is making me suspicious - IBUFDS_GTE2_X0Y3 vs IBUFDS_GTE2_X0Y2. Make sure the PCIe pins used are the same; you may need to do a bit of digging - possibly including switching from 'basic' to 'advanced' mode - to select the proper PCIe hard core site and transceiver sites. If the hard IP core example design works, look at the pinout log file to see what pins were used on that design as well as the implemented design to see which PCIe hard IP site was used, then make sure the other design is using the same pins and hard IP site. My guess is that it isn't, and as a result the transceivers on the FPGA aren't connected to the host and the link isn't coming up. It may be as simple as selecting the correct PCIe hard IP core instance, but you may need to change another setting or two as well.
0 Kudos
Reply
bhall0107
Adventurer
Adventurer
2,213 Views
Registered: ‎11-13-2018

@aforencich That's a good catch, thanks. It's strange, I thought that I had pasted the same IBUFDS constraint into the XDC, so maybe Vivado changed it automatically or something. Not sure if that's even possible, but I don't remember changing it. I'll take your advice and look into the hard IP site, transceiver, and pin placement. I'll let you know what I find. 

Thanks,

Brad

0 Kudos
Reply
bhall0107
Adventurer
Adventurer
2,160 Views
Registered: ‎11-13-2018

@aforencich @csattar , Here's what I found...

  • There is only one PCIe hard core in the Artix-7 that we are using, so it is definitely being placed in the same location.
  • The IO pin (sys_clk_p, sys_clk_n, sys_rst_n) constraints are being placed in the same location (E11,F11,M20).
  • refclk_ibuf is getting placed into IBUFDS_GTE2_X0Y2 for both designs even though Vivado automatically added the constraint to IBUFDS_GTE2_X0Y3 in the XDC for the 7 Series PCIe Integrated Block example.

The only differences in placement that I saw were that the MMCMs are placed in different locations along with various buffers that are created by the example designs. I assume this is part of the problem. Maybe the design doesn't satisfy the requirements in PG054 pages 187-188 and 197-200?

I don't have time to dig deeper into this. Example designs should work with minimal effort especially when targeting a Xilinx evaluation board. In IP Integrator, why is there no PCIe option in the board tab like there is for different boards (i.e. Kintex KCU 105, shown below)?

wtff.png

Does anyone know if there an easier way to implement PCIe with DMAs on the AC701? Maybe by using the 7 Series PCIe Integrated Block and connecting DMAs to that? I would need test or example drivers that we could develop for Linux and Windows. Are there other FPGAs (or evaluation kits) where this is easier to achieve?

I'd really appreciate any advice or assistance. Thanks!

0 Kudos
Reply
bethe
Xilinx Employee
Xilinx Employee
2,122 Views
Registered: ‎12-10-2013

Hi @bhall0107 

I would recommend building in the following:

- Select AC701

- IP Catalog - Selection DMA/Bridge IP

- Configure to your settings

- Right Click, Open example design

- Build

You shouldn't need to change any of the constraints.

If that is not showing up in device manager, can you give the machine a quick *warm* reboot?  The DMA core takes significantly longer to load into the FPGA, and in some hosts it is too slow, and misses enumeration.  On a cold reboot, the FPGA loses power and has to reload, but on a warm reboot, the system just re-enumerates.

If that doesn't work, if you could add a JTAG Debugger (on the Debug tab in core configuration, prior to open example design -- will need to open new example design) and see where the LTSSM is getting stuck?

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
bhall0107
Adventurer
Adventurer
2,099 Views
Registered: ‎11-13-2018

Hi @bethe, thanks for your response.

I followed the steps you listed "- Select AC701, IP Catalog - Selection DMA/Bridge IP, - Configure to your settings, - Right Click, Open example design, - Build" without making any manual edits to the XDC file. Here's what happened:
unconstrained IO.png

This is why I added constraints as described in my original post. 

I have tried warm reboots on XDMA example designs with constraints added, and nothing seems to change. That suggestion makes sense, and I'm not sure why it doesn't help. Do you have any ideas why it wouldn't? Are my constraints wrong? Maybe it has to do with the other axi_resetn assertion conditions (from PG054 page 185) : 

  • PLL within the Core Wrapper: Loses lock, indicating an issue with the stability of the
    clock input.
  • Loss of Transceiver PLL Lock: Any transceiver loses lock, indicating an issue with the
    PCI Express Link.

I do not see the Debug tab or the option for a JTAG Debugger on the XDMA core configuration (see below). I've seen it when targeting other boards, but not the AC701.

nodebug.png

Lastly, do I need to select anything in the Shared Logic tab? This post mentions something about it, but it's a little unclear... https://forums.xilinx.com/t5/PCIe-and-CPM/DRC-Failure-on-PCIe-IP-Implementation/td-p/918375 

Thanks for your time and support.
Brad

0 Kudos
Reply
bethe
Xilinx Employee
Xilinx Employee
2,024 Views
Registered: ‎12-10-2013

Hi @bhall0107 

After writing the post, I was running a new example just to be sure.  It does look like there are some issues in there, so I am having it run in hardware today, and will post as soon as we figure out what is going on.  For sure, we need to get the XDC issues fixed on the board files. 

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
bhall0107
Adventurer
Adventurer
2,016 Views
Registered: ‎11-13-2018

@bethe Thanks for looking into it. One thing I found was increasing the bitstream configuration frequency (BITSTREAM.CONFIG.CONFIGRATE) to 33MHz allows the FPGA to be configured before the host asserts the PERSTN signal. Now XDMA asserts the link up signal. The problem is that the host does not boot. It seems to be stuck before/on BIOS. Have you seen anything like this before? 
Hope that information is helpful.

Thanks again.

0 Kudos
Reply
bhall0107
Adventurer
Adventurer
1,937 Views
Registered: ‎11-13-2018

Hi @bethe,

Have you made any progress on this? We've been exploring other IP options, but would like to use XDMA on the Artix.

Thanks.

Brad

0 Kudos
Reply
nbradley777
Newbie
Newbie
1,835 Views
Registered: ‎03-06-2020

I too am interested in what you discover. I am having issues with getting XDMA to work for a Vivado project targeted at the AC701 board. I have ran into all the same problems as OP, using Vivado 2019.2.

 

Thanks.

0 Kudos
Reply
bethe
Xilinx Employee
Xilinx Employee
1,795 Views
Registered: ‎12-10-2013

Hi all,

 

Can you add these constraints to the top level:

###############################################################################
# Pinout and Related I/O Constraints
###############################################################################

#
# SYS reset (input) signal. The sys_reset_n signal should be
# obtained from the PCI Express interface if possible. For
# slot based form factors, a system reset signal is usually
# present on the connector. For cable based form factors, a
# system reset signal may not be available. In this case, the
# system reset signal must be generated locally by some form of
# supervisory circuit. You may change the IOSTANDARD and LOC
# to suit your requirements and VCCO voltage banking rules.
# Some 7 series devices do not have 3.3 V I/Os available.
# Therefore the appropriate level shift is required to operate
# with these devices that contain only 1.8 V banks.
#

set_property IOSTANDARD LVCMOS33 [get_ports sys_rst_n]
set_property LOC M20 [get_ports sys_rst_n]
set_property PULLUP true [get_ports sys_rst_n]

###############################################################################
# Physical Constraints
###############################################################################
#
# SYS clock 100 MHz (input) signal. The sys_clk_p and sys_clk_n
# signals are the PCI Express reference clock. Virtex-7 GT
# Transceiver architecture requires the use of a dedicated clock
# resources (FPGA input pins) associated with each GT Transceiver.
# To use these pins an IBUFDS primitive (refclk_ibuf) is
# instantiated in user's design.
# Please refer to the Virtex-7 GT Transceiver User Guide
# (UG) for guidelines regarding clock resource selection.
#

#set_property LOC IBUFDS_GTE2_X0Y3 [get_cells refclk_ibuf]
set_property LOC F11 [get_ports sys_clk_p]
###############################################################################
# Timing Constraints
###############################################################################
#
create_clock -name sys_clk -period 10 [get_ports sys_clk_p]
#
#
set_false_path -to [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]
set_false_path -to [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S1}]
#
#
set_case_analysis 1 [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]
set_case_analysis 0 [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S1}]
set_property DONT_TOUCH true [get_cells -of [get_nets -of [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]]]

set_clock_group -name async1 -asynchronous -group [get_clocks {sys_clk}] -group [get_clocks {clk_125mhz}]
set_clock_group -name async2 -asynchronous -group [get_clocks {sys_clk}] -group [get_clocks {clk_250mhz}]
#
#
# Timing ignoring the below pins to avoid CDC analysis, but care has been taken in RTL to sync properly to other clock domain.
#
#
##############################################################################
# Tandem Configuration Constraints
###############################################################################

set_false_path -from [get_ports sys_rst_n]

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------

View solution in original post

bhall0107
Adventurer
Adventurer
1,745 Views
Registered: ‎11-13-2018

Thanks @bethe, we were able to get the XDMA example design working on the AC701 board using these constraints. There were a few things that I wanted to mention though...

First, the example design did not work with Shared Clocking enabled. The host PC would not boot. The design did not meet timing and gave the following critical warnings:

with_shared_logic.png

 

Next, the example design worked with Shared Clocking disabled. I was able to boot the machine and use the example programs to verify that it was operational. The following critical warnings still appeared:

no_shared_logic.png

I commented out all of the constraints that caused critical warnings in implementation and everything still worked fine. It looks like they were meant for the 7 Series PCIe Integrated Block example design. Is that true? Should I edit them so they apply to the XDMA example design?

I am wondering which constraints you added actually made a difference. It seems like the 
set_clock_group constraints must have done something. I also noticed that set_property LOC IBUFDS_GTE2_X0Y3 was commented out. Was that for a specific reason? I see that the buffer was placed at site IBUFDS_GTE2_X0Y2.

Thanks again for your time and support on getting this to work. We're glad that we can move forward using XDMA.

Brad

 

0 Kudos
Reply
likewise
Contributor
Contributor
804 Views
Registered: ‎09-24-2016

Although I think you are up and running, you mention you have seen the link come up but the (x86?) PC hangs during booting.

I have seen this issue occur when the PCIe device class configured in XDMA (or PCIe block for that matter) is a Serial Controller (UART), and a x86 PC BIOS attempts to interact with it (which fails, because it does not fulfill that class).

Changing the class (to a memory controller) solves this issue.

 

0 Kudos
Reply