Timing Analysis

[terrybarnaby]

We are having problems with a Virtex6 VHDL application running on a 6vlx240t-ff1156-1 device on a custom board. It is a DSP application with FIR filters etc (no DSP blocks being used). It is an old design that used to run on a Virtex2 at 160MHz.

The main clock is 200MHz with a separte bus clock of 125MHz for the PCIE interface (clock domian crossing circuits are in place). The design is currently using about 20% of the slice resources and 80% of the BRAM's. It has appropriate constraints on the clocks including INPUT_JITTER and it passes timing constraints.

However, it is giving a large number of bit errors in the signal data stream. Playing around with the design affects things but does not make the problem go away (moves from I to Q signals or different bits are more/less affected). This does appear to be a timing issue to me. The 200MHz clock constraint is just met (about 30ps slack) other clocks have around 2ns slack. We have tried running the design on a -2 part. It is better but it still fails on this (even when constrained for a -1 part).

My question is, how reliable are the Xilinx tools in calculatiing timing and placing/routing accordingly ?

Any ideas on tracking down what is happening (as soon as I mod some bit of code to try and work out where the errors are occuring, the errors change).

[austin]

t,

Reliable?  It is our businedss.  It is our gaurantee.  It is, just eveything.

What is your system jitter?  I have seen cases where the slack was positive, but the system jitter was 1000 ps from too many strongly switching IOs and CLB's, and the peak min period jitter is then 500 ps, so you would need at least 500 ps of positive slack to meet timing.

What else could be causing your problems?  There are other reasons why you may be getting "bad data."

In any event, I would be submitting a webcase, and demanding my local FAE come help sort things out.

If it is a system jitter issue (or if you suspect it is a system jitter issue), can you move your IOs or some of your clocks to another phase (of the dsame clock)?  Best (least jitter) is the main clock is 270 degrees ahead of IOs switching (min system jitter point).  Can you switch half the logic on the 180 degree  clock?  Often going to a faster clock reduces jitter as the bypass networks become more effective (at the higher frequencies).

Any given part, is likely to be a -3 speedgrade part, marked as a -1 part, because we need to fufill orders, and when a product is reasonably mature (a year or so into production) it yields well, and all die are pretty fast.

If you are are below the 85C junction temperature (100C for I grade), and the voltage is within its specified range, and the system jitter is under control, and you have positive slack (and you have constrained all critical data paths) you will have no timing violations.

[terrybarnaby]

Thanks for the reply.

I don't know what our systems jitter is, any way to measure or determine that ?

I have assumed the tools define and use a typical value ?

There is almost no direct IO. There is a 4 lane PCIE, an FPDP port on a GTX, and a few LED's. Data path is FPDP through logic/BRAM's and out on the PCIE. Only the core DSP functionality runs at 200 MHz.

As far as CLB's, only 20% of the slices are used, however 80% of the RAMB's are in use. The 200 MHz crystal is a low jitter (1.35ps) LVDS direct clock. There are two clocks 200 and 125 (from PCIE) with half of the RAMB's in each clock domain.

Temperature is only about 45 degrees, core voltage is 0.995V +-1%. I would not have considered it highly loaded and I have assumed all of this should result in a low system jitter ?

We do see a noticable difference between a board with a -2 and a board with a -1 part we have. Also using the 13.3 tools gave a working bit file while the 13.4 tools didn't (I suspect

Thanks for the advise I will try and see if I can find out how to contact a local FAE ...

[austin]

t,

System jitter is something you must enter into the tools, as it can only be measured after the fact, and only guessed at before.

To measure, take the clock under suspicion and send it to a DDR IO  (called clock forwarding).  The DDR DFF has the top flop D connected to Vdd, and the bottom clock D connected to ground, so that on the rising edge, it outputs a 1, and on the falling edge it outputs a 0, thereby faithfully reproducing the clock at the IO pin,  From there, send it to a jitter measurement instrument (Tektronix has jitter measurement packages in their high end oscilloscopes, as does Lecroy, and Agilent).  Try to measure the peak to peak jitter, unfiltered.  It is 1/2 the peak to peak that makes the clock its smallest (fastest) and takes away from the slack.

I have seen 1000 ps and more of peak to peak jitter in some designs.  From your desription, this may not be your problem.

Since you have PCIe and GTx as well, how do you know you do not have signal integrity problems which is leading to a low bit error rate on these channels?

The temperature sounds OK (it is the junction, right?  and not the case...), and the voltage is proper.

Changing tools before you know what is wrong is troublesome:  I agree that if you change nothing, nothing will change, but if you change too much, you do not know what was fixed, and what is broken.  Rather, I would look at the unconstrained paths -- why would one version work, and antoher not work?  Most often it is the fault of a missing necessary constraint, or a constraint that is not set correctly.

If you cross clock domains, have you made sure that all clock crossings are properly synchronized with 2 or more stage synchronizers to aoid metastability occurences?

Hope that helps,

[mcgett]

Terry,

The description of your problem is highly indicative of one of the following conditions:

• a design with incorrect or missing timing constraints
• a design with latches
• a design with asynchronous or clock domain crossings that are not synchronized

In your original post you said that you are moving from a Virtex-II to a Virtex-6 design.  This is a big leap, so there is likely a lot of code changes and some of the above items can be missed in the design conversion.

[terrybarnaby]

Austin, Thanks for the info. I assume I will need to tap off the clock from a few different phyisical places in the design to get a reaonable measumenent ? Is there a conventinal method of how to do this and how to route the tap (use fpga_editor) ?

We have done a fair bit of testing of the PCIe and GTx interfaces. The bit error often occurs in just the I or the Q data streams and look to be near the end of filtering (the bit errors do not look like they are filtered). This rules out the input side and due to the fact the I/Q data is output in 24bit format over a 32bit interface over PCIE rules out the PCIe side. It can be a high error level (30 in 1024).

Yes, I agree with the changing tools comment, but we are starting to get deparate ( :) ) and trying to get information on what affects the issue to help try and pin it down.

There are only two clock domain crossings in the data path, (from 125 to 200 and 200 back to 125) and these do have 2 stage synchonizers.

Thanks for the input.

[terrybarnaby]

mcgett, we have inheritted the Virtex2 design and there could be issues in there. So far there have been minimal code changes for the Virtex6 (RLOC attributes mainly, athough due to the issues we have changed the RAMB usage to use their internal output registers to improve the timing of these). It is a fairly conventional design and we were hoping that the top level clock timing constraints on the 200MHz and 125MHz clocks would cover all the paths within that code. I don't think any latches are used, but will have to look in detail.

The clock domain crossings are synchronised, but I will have to check that the registers are not being optimised out.

Thanks for the input.

[mcgett]

> There are only two clock domain crossings in the data path, (from 125 to 200 and 200 back to 125)

> and these do have 2 stage synchonizers.

Two stage synchronizers will work well for single bit control paths, but if you are passing parallel data back and forth than you need a FIFO to ensure that the entire word crosses the boundary at the same time.

>  minimal code changes for the Virtex6 (RLOC attributes mainly,

RLOCs imply that there were very tight timing issues in the original design.  These should be examined very closely to determine why they are there as 125 MHz and 200 MHz clocks should not be a major problem to handle in the Virtex-6.  LOC constraints for the IOs, PCIe and GTX pins should be everything that you need for placement constraints.

[terrybarnaby]

The two stage synchroniser is just for a data valid signal. The actual data is latched on the incoming data valid signal and an output data valid signal is generated by the two stage synchroniser. The data rate is quite a bit less than either clock rate.

I think I have found my problem. As everyone has mentioned first "incorrect or missing timing constraints" :) There was a constraint on the PCIE clock which is being used as a master clock for IO in the system. However the timing contraint was not being passed through the PCIE block and so although it appeared on timing lists it was only being applied to circuits in the PCIE block and not the user circuits using the clock. I only found out about this after creating a timing report with the unconstrained nets reported using "trce" with the "-u <n>" option. I had assumed that as the tools had stated no timing errors then all timings were ok and all clocks had constraints. I had assumed the unconstrained nets would have been those that had been specifically marked as unconstrained.

The fact that there appeared to be issues in the 200MHz domain that moved to the 125MHz domain was probably due to too low a value of SYSTEM_JITTER and misled me into looking at the 200MHz side. Once I increased the SYSTEM_JITTER value, it probably helped the 200MHz side but  messed up the unconstrained 125MHz side. Amazing it worked at all with a lot of the 125MHz unconstrained, the placement/routing algorithms were making good choices  !

As a bit of feedback, it would be nice if the tools reported errors or at least warnings on any unconstrained clocks in the main timing reports and summaries (ones that have not been marked as unconstrained). Also the "Timing Closure User Guide" could make more of this, "check first the unconstrained list" ... The manual could also give more information on setting SYSTEM_JITTER bringing this more into the main requirements.

Anyway, thanks for your replies and info, it has helped.

[austin]

t,

Thanks for letting us know what was happening. And, we appreciate your suggestions.