06-21-2019 03:34 AM
06-22-2019 07:27 PM - edited 06-22-2019 07:32 PM
Ok, here is what’s got me bothered.
1) Vivado assumes all clocks are synchronous.
2) Xilinx FPGAs have features that ensure some clocks are synchronous (eg. all clock outputs of an MMCM are synchronous with each other – by design).
3) However, (and here’s the rub) what if there are features of Xilinx FPGAs that de-synchronize clocks? Do the Vivado tools know about these de-synchronizing features and will Vivado warn us about them? -or, is it up to us to keep a list of the de-synchronizing features, and a list of the not-synchronous (ie. mesochronous) clocks they produce, and (without help from Vivado) ensure that we don’t do things (like direct-crossing of data) between not-synchronous clock domains?
For example, in <this> post, @avrumw seems to indicate that if we route a base clock to two separate MMCM then the output clocks of MMCM1 will not be synchronous with the output clocks of MMCM2. I suspect this is true only if we turn OFF the “Phase Alignment (to input clock)” feature of one or both MMCM. However, my testing shows that whether or not the “Phase Alignment” feature of both MMCM is ON or OFF, Vivado v2018.3 issues no warning (timing analysis or otherwise) when I attempt a direct-crossing of data from the domain of a clock-output from MMCM1 to the domain of a clock-output from MMCM2. So, is this one (of many?) clock de-synchronizing features of Xilinx FPGAs that the tools will not warn us about?
06-26-2019 11:01 PM
06-27-2019 11:47 AM
I have made an edit to the refrenced post - my original statement about how "unkown" the phases between different MMCMs can be wasn't right.
The reality is that many output clocks from different MMCM will be in phase plus or minus some phase uncertainty based on the characteristics of the MMCM (a couple of hundred picoseconds). Unless the clock speeds are really high, these clocks can be considered synchronous, and Vivado will time them correctly, including taking into account the phase uncertainty through the different MMCMs.
The cases where MMCM outputs are completely out of phase are the result of frequency division (or multiplications that are not integers). In this case, there are multiple different possible phase relationships depending on the ratios involved. For example, if you have outputs that are 1/2 the frequency of the input clock on one of the outputs, there are two possible phase relationships between these corresponding outputs on two MMCMs - in phase, or 180 degrees out of phase. Again, depending on the frequencies involved, these can normally be considered "synchronous". The only question is if the tool figures out the correct relationship. As an example, if you have two MMCMs taking in 100MHz clock and each putting out a 50MHz clock, what does Vivado determine as the Requirement on the paths between the two 50MHz domains? If it says the requirement is 20ns, then this is not correct (assuming what I said about the MMCMs is correct) - since the two clocks can be 180 degrees out of phase, the requirement should really be 10ns (one 100MHz period).
As far as I know, all other mechanisms of managing clocks internally will result in correct timing analysis. Even if you do "non-recommended" clocking (like crossing between clocks that are based on the same input, but one uses a BUFR and the other uses a BUFG) they will be timed correctly - they may have bad hold time issues, but the tool will understand them and attempt to fix them (and depending on the frequencies involved it may even succeed).
I think this is true of any clock that can be traced back to a single input pin - with the possible exception of clocks from the GTs (which are both more or less traced back to the REFCLK of the GT); depending on the programming of the GT, RXOUTCLK and TXOUTCLK can be asynchronous or mesochronous. In the 7 series devices, though, these are considered "primary clocks" - in UltraScale, I think the tools will do automatically generated clocks from the GT REFCLK to *OUTCLK, and here you may be able get into some troubles...
06-27-2019 12:16 PM - edited 06-27-2019 12:19 PM
Thank you for prompting me to reread UG949. This is an awesome document that I must remember to reread and read-more-carefully frequently.
I just read the June 26, 2019 (yesterday!) version of UG949 and I find that the following text from page-120 is new:
IMPORTANT: To ensure safe timing between parallel BUFGCE_DIV cells where the BUFGCE_DIVIDE property is set to a value greater than 1, both buffers must use the same enable signal (CE) and the same reset signal (RST). Otherwise, the divided clocks might become phase shifted from one another in hardware, which is not reported by the Vivado tools.
Aha! I understand this to mean that if you don’t heed this IMPORTANT warning then the BUFGCE_DIV output clocks will be mesochronous with other clocks in the design.
I also find the following text on page-118:
IMPORTANT: If the CDC paths are between clocks that originate from different MMCM/PLLs, the clock insertion delays across the MMCMs/PLLs are more difficult to control. In this case, Xilinx recommends that you treat these clock domain crossings as asynchronous and make design changes accordingly.
Again, aha! Sounds like mesochronous clocks are being created?
From UG472 July 30, 2018, page 110 says:
BUFR Alignment: When using the built in divide capability of the BUFR (shown in Figure A-6 and Figure A-7, the clock must be stopped at the BUFMR and reset signals applied to the BUFRs, to align the BUFR divide counters across the multiple clock regions.
Again, again, aha! -more mesochronous clocks.
So, now we know. There appear to be valid clocking architectures in Xilinx FPGAs that (from a single base clock) will produce mesochronous clocks. Vivado will not warn us about these mesochronous clocks. Further, Vivado will ignore the fact that these clocks are mesochronous and perform timing analysis assuming these clocks are synchronous.
Xilinx, please, no pussyfooting around this issue. Let’s not call it “not reported by the Vivado tools” or “difficult to control” or “BUFR Alignment”. Plain and simple, the FPGA is producing mesochronous clocks. Please give us a list of the ways this can happen and plainly tell us that it is our responsibility to identify and handle these situations appropriately – because Vivado is not going to help us.
06-27-2019 12:56 PM - edited 06-27-2019 12:57 PM
I think we need to be more careful with the word "mesochronous". It doesn't simply mean "not aligned" - it's more than that.
Two clocks that have a known or bounded phase difference between them are "synchronous" - at least from the static timing point of view. For example two MMCM outputs where one goes through a BUFH and the other through a BUFG, these clocks aren't really mesochronous - at least at reasonable frequencies. They are still synchronous but with some phase uncertainty.
The (or at least my) definition of mesochronous is when the phase uncertainty gets high enough so that it is impossible to deal with them synchronously. So two 100MHz clocks with a clock skew of +/-1ns are not mesochronous - the tools can deal with 1ns of clock skew at this frequency (its expensive in terms of routing for hold time fixing, but doable). Take that same +/-1ns clock skew but on 400MHz clocks, and now it is impossible to cross between them synchronously - fixing a 1ns hold time violation at fast PVT requires roughly 3ns of delay at slow PVT, which is larger than your 2.5ns period, so these clocks are mesochronous.
I guess the same can be said for some of your BUFGCE_DIV examples - the phase uncertainty is measured in increments of one base clock. But the reality is that you can still work with these clocks more or less synchronously - you just need the correct timing exceptions. For example, if you have a 300MHz input clock going through two different BUFGCE_DIVs doing divide by 3 then you can have any of 3 possible phase relationships between them, 0, 1/3 and 2/3. The tools won't time this correctly - they will always assume they are 0. But even if they are 2/3 or 1/3, you can still do synchronous clock crossings between them - there is 3.33ns for the clock crossing, which is more than enough. So, if you constrain the paths correctly - say using set_max_delay 3.33, then the path will be correctly timed at 3.33ns. So are these mesochronous? Not really - they are just improperly timed...
07-29-2019 10:00 AM
-adding to the list of situations that may not be timed properly by Vivado is Avrum's comment from <here> that says:
WARNING! If you have two different outputs of the same MMCM with different phase shift values (with paths between them), you must NOT use LATENCY mode. In latency mode, the edge determination for paths between two clocks with different phase shift values is incorrect, and hence will result in incorrect timing analysis.