cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
gongjian1990
Visitor
Visitor
16,431 Views
Registered: ‎03-14-2012

VC707's PCIe doesn't work in ASUS Z87-Pro or Z87-Plus motherboard

I found a problem when using VC707 board in our newly bought computer with ASUS Z87-Plus motherboard. After Vritex-7 FPGA programmed with the example design and the system rebooted, I can't find the board using "lspci" in Linux. However, the ML605 board works well in that motherboard(even in the same PCIe slot) and can run at Gen2X4 mode or Gen1X8 mode.

We changed the mother board to ASUS Z87-Pro, but it doesn't recognise VC707 board as well. While the ML605 board works well in this Z87-Pro motherboard.

 

I finally got the motherbaord recognized the VC707 board with the bottom PCIe slot, but VC707 can only work in X2Gen1 mode. However the ML605 works in X4Gen2 well.

 

What is the difference between V6 and V7. It's strange that the mother board doesn't recognize VC707. 

BTW: The VC707 can work in X8Gen2 in an old ASUS motherboard in our lab. So I'm sure the FPGA configuration is right.

0 Kudos
6 Replies
markcurry
Scholar
Scholar
16,251 Views
Registered: ‎09-16-2009

Did you ever get resolution on this?

 

We've got a very similar setup, with similar failure modes.

 

We've got a KC705, which boots and operates fine on a variety of dell motherboards (H87 I think).

 

On a new Asus Z87 based board, the kc705 doesn't work - (same KC705 board that worked in the Dell's).

 

I get various failure modes.  Sometimes the board's not found at all (lspci doesn't show it.).  Sometimes lspci can detect it, but the BIOS/linux is not able to map the BAR. (Even if I reduce the BAR size to very small values - 64 KB).

 

In ALL cases the FPGA shows a good PCIE clock, and "pcie_user_lnk_up" shows the link is up.

 

I'm just beginning to debug this when I found this post.  Seems to match my failure mode.

 

I plan on using Answer record 56616 to help debug.  But that doc seems to focus on problem's achieving L0 state.

Since I get a good pcie_user_lnk_up, I think the board's succesfully reached L0

 

Thanks,

 

Mark

0 Kudos
markcurry
Scholar
Scholar
16,179 Views
Registered: ‎09-16-2009

 

FYI we've found some resolution with this problem.

 

This ASUS board will not MAP a BAR more than 32MB.  Ours requires 256 MB.

 

We're looking at a Gigabyte motherboard with similar specs (GA Z87X).  Similar problems - but this one will map a BAR size of 64MB successfully, but none larger.

 

I've contacted ASUS, but don't expect much help from them for such a custom card.  But we'll see..

 

I don't know if I can go in an manually re-enumerate the bus (partial or whole thing) after the BIOS has done its thing.  There's a pretty large nest set of PCIE bridges / and/or switches along the way...  Heck I don't even know how to get access to the low - level PCIE config settings....

 

--Mark (who's slowly learning WAY to much about "grey box" PC architectures...  Blech...)

 

 

 

0 Kudos
markzak
Explorer
Explorer
16,163 Views
Registered: ‎12-01-2010

I've been burned by my own issues with Motherboards and PCIe implementation, mostly in the performance & throughput arena.  The sad truth is that regardless of your PCIe design, you have NO CONTROL over the BIOS implementation and execution, let alone the motherboard  layout, processor busses, switches, etc.

 

 If you are designing products that will go into a variety of motherboards, you MUST design for the lowest common denominator.  In your case, it's a BAR that's 32MB.    There's no way around it.  Mobo manufacturers will rarely release new firmware to fix a problem if you're the only one that's complaining about BAR size.

 

 The best thing you can do in an optimal setting is research your motherboards & PCs ahead of time, BEFORE you start your PCIe design.  Ideally, you should try to pick a motherboard with limited peripherals, few PCIe slots thereby limiting or eliminating bus switches, copious amounts of memory, and a processor that meets your design needs.  It should be relatively trivial to find out the PCIe performance/limitations of the BIOS and motherboard online.

 

 

Check out this block diagram.Xeon E5 standard motherboad block diagram.png

 

This is from a Gigabyte GA-6PXSVT Motherboard, and is part of their manual.  It nicely illustrates the PCIe busses and their connection to the Xeon E5.

 

As they say, an ounce of planning is worth a pound of troubleshooting.

 

markcurry
Scholar
Scholar
16,158 Views
Registered: ‎09-16-2009

Thanks for the tips. 

 

The trouble for me is this type of stuff isn't documented, or easy to find at all.  I only figured out the 32 MB BAR limit for this ASUS board by trial and error.

 

We did research on the PCIe solutions and settled on the architecture (PLX PCIe switch with both the GPU, and our device sitting on the same switch).  But this BAR limitation stuff came out of nowhere, and my Google Fu just doesn't seem to be up to snuff.  I can't find much.

 

--Mark

0 Kudos
markcurry
Scholar
Scholar
15,423 Views
Registered: ‎09-16-2009

 

We've resolved our problems here.  Detailing the problem/fix here so that hopefully others may benefit.

 

Our problem was a stupid bug (aren't most of them) in how we were resetting the FPGA.


We're actually using a KC705, not the VC707 that the OP is, but the problem looks the same in any event.

 

For us, we had the KC705 plugged into the PC host.  We were, however usually, independently powering the KC705 with the external power supply.  We did this to allow us to power up the FPGA first, and configure it with impact .  Without doing this, the FPGA was configuring too late - past the PCIE spec initialization requirements, and the device wasn't always recognized by the BIOS/OS.

 

We've got some of our usual reset logic in place inside the FPGA. This logic insures that all reset requirements are satisfied - durations required, and stepping, synchronization, etc..  Most of the time this is all started by an "or" of a configuration time one shot, and a pin reset.

 

All well and good.  Most of the time the pin reset's never used.  The FPGA is self initializing, and deriving it's reset from the one shot is fine.

 

Problem was we didn't enter the PCIE_PERSTN connector into this reset logic.

 

So, by independtly powering the KC705, and configuring it before the host was powered, the FPGA configures itself, and does it's reset scheme.  Problem is there's NO clock on the PCIE_REF pin at this time.  And since this clock goes to all sorts of non linear things (PLLS, GTX's, etc.), problems can occur. 

 

When the host finally does power up, we don't reset the KC705 again - missing some of the requirements.

 

As with all reset/initialization problems - this can present itself in very weird ways.  I've no idea why for us the pass/fail correlated so much with PCIE BAR sizes, but it did - reliably. 

 

Anyway the fix for us was simple - adding the PCIE_PERSTN connector into our reset logic.

 

I'm usually almost excessivly careful with reset requirements - sometimes even to the annoyance of some of my co-workers.  But I got bit again.  And these types of initialization/reset problems are the devil to find and fix...

 

Regards,

 

Mark

gongjian1990
Visitor
Visitor
15,149 Views
Registered: ‎03-14-2012

Finally I solve the problem. I found Xilinx has an answer record about this issue. See: http://www.xilinx.com/support/answers/51135.html

The ASUS motherboard uses the intel Z78/Z77 chipset. It seems that the chipset is not working well with the 7-series IP core. 

The answer record says we can "To work around this issue, set TX_RXDETECT_REF to 3'b011 in "gt_wrapper.v". By default, this parameter is set to : 3'b100." and I tried this out then my ASUS Z87 motherboard can detect VC707.

 

Regards,

 

Jian

0 Kudos