Hello again and welcome to the latest RF Data Converter Blog.
So far we’ve learned about the RF Data Converter Software drivers here and took a deep dive into the RF Analyzer that allows you to debug RF-ADC and RF-DAC on any device on any board (Part One and Part Two).
I think now is a good time to discuss a topic that is very important to many customers using RFSoC: The requirement to achieve latency alignment across multiple tiles either on a single device or indeed across multiple devices. We refer to this requirement as Multi-Tile Synchronization.
Multi-Tile Synchronization is key to enabling Massive MIMO, Beamforming, and phased array radar applications.
For example, in beamforming the idea is that rather than simply broadcasting energy in all directions you use an array of antennas that can be steered to transmit radio signals in a specific direction. In this technique, each antenna element is fed separately with the signal to be transmitted. The phase and amplitude of each copy of the signal is then added constructively and destructively in such a way that they concentrate the energy into a narrow beam or lobe.
It follows that it is necessary to use many data converters to build the array and there is a requirement to have latency alignment across all channels in the antenna array.
Let’s further expand on the idea of latency alignment as it applies in this case. You’ll hear about latency alignment and determinism.
Latency alignment means that the relative latency across all channels is the same, determinism means that the total latency across all channels is consistent at every start-up. In some case there is a need to have both determinism and alignment.
At start-up of the RF Data Converters, converters in a single tile are always aligned, but there is no guarantee of deterministic latency. In the multi-tile system there is no guarantee that you will have deterministic latency or even latency alignment across tiles. This means that we have to provide a mechanism to align these tiles. This is implemented inside the IP and it is managed by API calls in the software driver.
The easiest way to understand how Multi-Tile Synchronization really achieves alignment is to first understand the sources of the alignment uncertainty we are trying to eliminate. We’ll go through this in detail, but it is really important to understand something before proceeding.
Enabling this functionality in the IP and using the software API to get the tiles to align is of course mandatory, but all this mechanism does is provide digital alignment across tiles. There are PCB and Clocking rules that must also be followed. These are outlined in the PCB Design User Guide.
With that in mind let’s talk about the main sources of latency uncertainty you will encounter. Take a look at the diagram below.
I’ve numbered the various sources of latency mismatch across the tiles:
The RF-ADC or RF-DAC tile clock input needs to be aligned, any mismatch here will mean that the converters do not sample at the same instant. This can never be corrected internally. For this reason the traces must be delay-matched on the PCB.
If the Tile PLL is used to create the sample clock, then between two tiles there is no guarantee that the output divider on the PLL will have the same phase. This is because we don’t control when it comes out of reset at the start-up. We need a way to synchronously reset all of these dividers across tiles so that they are aligned.
Similarly, the digital portion of the RF-ADC and RF-DAC tiles runs on a divided version of the converter sample clock. Across tiles, there is no guarantee that these dividers come out of reset with the same phase. These dividers need to be brought into a common reset state.
The FIFO that passes the data safely between the Tile and the PL Fabric can have either M or M+1 read cycles of latency depending on when the read enable is asserted versus the write. This requires a means of correction to achieve the tile synchronization.
To address the above concerns we have a solution that enables synchronization across tiles. It is implemented in the IP and has its own set of API calls in the RFDC driver to control. The key to this scheme is that we borrow the SYSREF concept that is used by JESD204B. We are going to use the SYSREF as a common timing reference for the system. There are some rules for SYSREF which are all covered in (PG269) and (UG583) and I’ll relate them to the synchronization procedure as we go. There is a need to provide a SYSREF to both the tiles and to the PL Fabric (Known as Analog SYSREF and PL SYSREF respectively). It will become clear why shortly.
But first, how should you get started? Before we look at the solution there are some PCB concerns to be aware of.
Your ADC and DAC tile Sample clocks must all be phase aligned and arrive at the tile clock input at the same time. Also, DAC output paths and ADC input paths must be delay matched. Remember the solution is only providing digital alignment here, mismatch on the clocks or data lines will show up as residual skew after you’ve completed the tile synch.
The Analog SYSREF and PL SYSREF signals must be routed to the RFSoC such that they both arrive at their respective inputs at the same time. (This is important, again all will be revealed later on.)
In the design you must enable MTS for the tiles you wish to synchronize in the IP.
Remember that the lowest order numbered DAC and ADC tiles always must be in the synchronization group.
The software application must include the API calls to do multi-tile synchronization at run time.
Another good step is to set the log level of the metal log to DEBUG for testing MTS in your setup. The metal log gives a lot of detail about what is happening during MTS and it is invaluable when debugging issues with MTS.
In your SW application, you will need to declare a structure for ADC and DAC synchronization groups.
You will need to initialize and set these structures up to do MTS.
In the most simple case you just need to specify which tiles need to be synchronized and call the Multi-Tile synchronization function XRFdc_MultiConverter_Sync to get the IP to align the tiles:
So what happens under the hood when the API is run?
This is where the metal log is really informative.
The first thing that happens is that the SysRef is distributed to all tiles to be synchronized. Then we capture the SYSREF with the analog sample clock in the tile using a delay tap chain (DTC). If you have the PLL enabled in the tile it will also safe capture it with the PLL VCO as well.
You can see in the log that it starts in the middle of the delay tap chain at tap 64 and sweeps to find the ideal tap so that the SysRef is in the centre of the sample clock period.
If you look at the DTC sweep you will see 0’s. This is the stable part of the clock period, bounded by 1/2/3 which represent transitions. You will see the scan places a # and a *. The hash represents where it started and a star is the DTC code it settles on. It will use the code it picked to seed the DTC for the next tile. You can see in tile 0 it finds the ideal code at tap 44 and then begins in tile 1 with code 44 before trying a few codes and settling on the code 47.
This is why we require that the SYSREF signal must be a high-quality, free-running, low-jitter square wave. If it is noisy then we are going to get a mismatch in where we capture it which can lead to misalignment across the tiles.
Once we are safe capturing it in the tile, we use the SYSREF to synchronously reset all of the dividers in the digital part of the tiles. This is the reason why we ask that the SYSREF frequency be an integer sub-multiple of the global clock divider, GCD(DAC_Sample_Rate/16, ADC_Sample_Rate/16) as well as any PL side clock that samples it.
Once this divider reset happens there is effectively a common clocking across all of the tiles. Everything internal to the tiles is aligned. If you go back to the picture above showing the sources of latency misalignment, you can see that we’ve taken care of items 2 and 3 here.
The job is still not done because we need to account of the inherent mismatch between tiles that comes from the dual clock FIFOs. How should we do this?
First the PL User SYSREF must be captured in the PL Clock domain and if it is different, the AXI-Stream clock domain. This shows again why the SYSREF must be an integer submultiple of all PL Clocks. So now the SYSREF is getting safe captured in the PL clock domains.
I mentioned earlier that I would make clear why there was a need for an Analog tile side SYSREF and a PL user SYSREF and why we wanted them to arrive at their respective inputs at the same time.
The next step of MTS is to effectively take the PL user SYSREF and The Tile SYSREF and compare the time of flight for each of them across the tiles.
Because they are aligned at the device package balls, they are safe captured and arriving at either side of the FIFOs at the same time. So any mismatch in this “Time of Flight” or relative latency across tiles can only come from the FIFO. In this case we use the IP to insert what we call a marker bit into the FIFO. It is used to stop the marker counter on the read side of the FIFO. Then the marker counters are compared. We can then adjust the read pointer of the FIFO so that all FIFOs are matched.
You will see the metal log show you the marker counter reads and any adjustments that get made.
Finally you can make a report on the MTS adjustments.
After the tile synch is performed you can observe the latency alignment in hardware.
The capture below shows the result for a single tone input to all 8 ADCs on a 28DR after MTS is performed.
Provided all of the guidelines are followed you should achieve alignment that is within our spec of +/-1 T1 clock period.
In practice you get residual misalignment in the sub T1 range. As discussed earlier on, this mismatch really comes from the PCB traces for the analog I/O and the tile input clock.
What about deterministic latency?
At the beginning we said that in some cases it was necessary to have both alignment and determinism from power up to power up. The capability to do this is built into the MTS APIs.
One of the members of the data structure for MTS is Target_Latency. This can be set to give a target that the IP will adjust to so that you always get the same latency at the FIFO.
The procedure to do this is to set the target latency to 0 and observe the FIFO with the largest latency measurement, add a margin to it, and set this value as the new target.
This margin is expressed as a number of sample clocks. For the RF-ADC tiles, this value must be a multiple of the number of FIFO read-words times the decimation factor, and for RF-DAC tiles a constant of 16 is shown to be very robust.
Remember, the default behavior of MTS is to align the tiles, so if the target is set too low the metal log will warn you that it cannot meet it and it will just synch the tiles.
In this blog I’ve tried to explain how MTS works and to relate it to the metal log. I hope this blog gives you more insight into the Multi-Tile Synch solutions and that it helps you to pull together information from the IP product guide, the PCB guide and your own experience of using this feature as well as helping you to understand the messages coming from the metal log.
There are many other capabilities in the metal log in terms of error reporting which will perhaps form the basis of another blog on debugging MTS failures in the future.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.