UPGRADE YOUR BROWSER

We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Adventurer
Adventurer
1,765 Views
Registered: ‎01-21-2012

Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

I have been unable to synchronize the RFSoC ADCs when the NCO is enabled. This is my procedure. Please correct / comment:

1. Enable all clocks and verify they are correct: Tile 224, 225, 226, and 227 RF clocks at 3932.16 MHz, Tile 228 SYSREF Clock at 6.144 MHz, FPGA PL SYSREF Clock at 6.144 MHz

2. Load the Bitstream. The RF Data Converter block has the ADC Tiles and DAC Tile 0 Enabled, Multi-Tile Sync Enabled for the ADC and DAC Tiles, the Decimation Setting, NCO Frequency,  Mixer Mode, and other parameters set to the desired values. Here is a screenshot of the core configuration GUI:

 RFSoC ADC GUI Setup.png

3. After the bitstream is loaded, I run the software setup, I first run XRFdc_Reset() on all Tiles, then , then XRFdc_StartUp() on all Tiles, then XRFdc_SetupFIFO() on all tiles.

4. Next in the code, I setup the Mixer Settings for all ADC Tiles:

 

Mixer Settings Code.png

5. I then run the Multi-Tile Sync for the ADC

ADC MTS Code.png

6. I get an MTS good result:

ADC MTS Result.png

The problem is that the ADC outputs are not synchronized and consistent from run to run. If I run this procedure ten times, I will get ten different phase deltas between the signals out of tile 224 and 225. What am I doing incorrectly in the sequence? Is there a document other than the RF Data Converter User Guide that has step-by-step instructions for synchronizing the ADCs with an active NCO?

0 Kudos
18 Replies
Moderator
Moderator
1,712 Views
Registered: ‎04-18-2011

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

Hi @kendrick

Is this your own board or ZCU111?

Have you followed the SYSREF guidelines on the PCB? 

Can you try to run the MTS example from the SW driver? 

have you tried getting it to run and not setting a target latency?

Are you able to check the metal log to see what is happenning at each step. 

It is a good idea to enable the metal log at least for error and info messages

 

 

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Adventurer
Adventurer
1,699 Views
Registered: ‎01-21-2012

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

It is the ZCU111.

Which SW driver and which MTS example are you referring to?

I will try not setting target latency and enable/check the metal log.

Will report back soon...

0 Kudos
Moderator
Moderator
1,690 Views
Registered: ‎04-18-2011

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

if you look in your SDK install

Say:

C:\Xilinx\SDK\2018.2\data\embeddedsw\XilinxProcessorIPLib\drivers\rfdc_v4_0\examples

you will see xrfdc_mts_example.c 

import that into an empty application in your SDK project and give it a try. 

If you set a target latency and subsequently get a latency that is higher your MTS will error out. 

To set the target properly you have to run with the target set to 0, find the highest latency and add a margin to it.

I would try without a target and enable the metal logging for infos and errors. 

I attached the example to the last post. 

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
Adventurer
Adventurer
1,683 Views
Registered: ‎01-21-2012

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

The missing piece was the command to reset the NCO Phase:

 

Reset NCO Phase.png

I used the PL event instead of the SYSREF event to reset the phase because once I kick off the SYSREF clock from the LMK04208 onboard the KCU111 I don't really have a good way to pause it. With the PL event (on the RTS bus), I can create a one-shot type event in the fabric.

If using SYSREF, is the correct procedure to disable the signal until after the registers are setup and before the MTS procedure is performed?

0 Kudos
Moderator
Moderator
1,655 Views
Registered: ‎04-18-2011

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

Hi @kendrick

I need to check that reset NCO phase is dependent on the tile event. 

We expect that the SYSREF is running for the duration of MTS and can then be gated exeternally so that we could do a synchronous update to NCOs etc. 

(2018.3 will enable fast NCO updates)

Let me double check it. 

 

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
0 Kudos
Visitor bmarcotte247
Visitor
1,405 Views
Registered: ‎05-22-2018

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

I am having trouble synchronizing all ADCs with NCOs Enabled too.   How are the NCO Phase's synchronized across multiple tiles? 

My current steps are:

1) Call XRFdc_MTS_Sysref_Config to disable SYSREF

2) Call XRFdc_SetMixerSettings to set up for complex and NCO enabled and eventSource set to SYSREF

3) Call XRFdc_ResetNCOPhase

4) Call XRFdc_Startup

5) Call XRFdc_MultiConverter_Init

6) Call XRFdc_MTS_Sysref_Config to enable SYSREF 

7) Call XRFdc_MultiConverter_Sync

My SYSREF runs continuously, so I disable it using API call before setting up the mixers and NCO.  I suppose I have a race condition when I turn it back on before calling MultiConverter_Sync.  What I am seeing is that the Phases aren't synchronized across tiles.  Any suggestions?

Thanks!

 

0 Kudos
1,048 Views
Registered: ‎02-15-2019

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

Hi Kendrick,

I'm trying to do something similar. I noticed you changed the event source to synchronize the NCO phase of the mixers across tiles from SYSREF to PL. One thing that concerned me about this approach was that it seems that the PL event needs to be synchronous to the clk_adcX clocks coming out of the RF IP core. Are you just double-synchronizing your PL event, which I'm assuming is on your axi_aclk, onto each adc clock (adc_clk0 - adc_clk3) and constraining them as asynchronous? Or did you have another strategy for issuing the PL event synchronously to all 4 adc tiles?

Thanks!

-Sean

0 Kudos
Participant sasquatch
Participant
630 Views
Registered: ‎07-22-2018

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

Hello,

I'm getting an XRFDC_MTS_TIMEOUT error returned from XRFdc_MultiConverter_Sync() and would like to check the metal log.  Does anyone know how to enable the metal log and if it's enabled, where the actual log is stored?  I googled and found surprisingly little info on this.

Thanks!

0 Kudos
Moderator
Moderator
623 Views
Registered: ‎04-18-2011

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

Hi @sasquatch 

I cover RF and can't speak for other embedded SW folks but I normally use a handler. 

Something like this:

void my_metal_default_log_handler(enum metal_log_level level,
const char *format, ...)
{
char msg[1024];
char msgOut[1048];
char *outPtr;
int i;

va_list args;
static const char *level_strs[] = {
"metal: emergency: ",
"metal: alert: ",
"metal: critical: ",
"metal: error: ",
"metal: warning: ",
"metal: notice: ",
"metal: info: ",
"metal: debug: ",
};

va_start(args, format);
vsnprintf(msg, sizeof(msg), format, args);
va_end(args);

//replace single \n with \n\r
outPtr = msgOut;
for(i=0; i<1024; i++) {
// if /n/r or /r/n combo
if ((msg[i] == '\r' && msg[i+1] == '\n') ||
(msg[i] == '\n' && msg[i+1] == '\r')) {
*outPtr++ = msg[i++];
} else if(msg[i] == '\n') {
//if first char in string is \n, then remove
if(i==0) {
continue;
} else {
*outPtr++ = '\r';
}
}
*outPtr++ = msg[i];
if(msg[i] == 0) {
break;
}
}
//if line doesn't end with \n\r, then add it
if( (msg[i-1] != '\n') && (msg[i-1] != '\r') ) {
*(outPtr-1) = '\r';
*outPtr++ = '\n';
*outPtr++ = 0;
}

if (level <= METAL_LOG_EMERGENCY || level > METAL_LOG_DEBUG)
level = METAL_LOG_EMERGENCY;

xil_printf("%s%s", level_strs[level], msgOut);
}

To use this the following is needed:

You need to add a prototype at the top of the file

void my_metal_default_log_handler(enum metal_log_level level,const char *format, ...);

Then you have to nominate the handler and the messaging level in the structure when you initialize. 

struct metal_init_params init_param = { \
.log_level = METAL_LOG_DEBUG, \
.log_handler = my_metal_default_log_handler, \
};

Once I do this I get the metal log output into the console / terminal over the UART

I hope this helps. 

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
Participant sasquatch
Participant
584 Views
Registered: ‎07-22-2018

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

Hello, and thanks for the quick reply!  Your post did help me find that there is actually a GetLog() command in the rftool example source, which returns a string from the metal log.  I added a printf statement to the rftool command parsing loop that prints out the rxBuf and txBuf for incoming commands and their responsese, respectively, to the UART output.  I also realized that I had called XRFdc_DynamicPLLConfig() with incorrect arguments.  I had set up my clocking with an external PLL ref clk of 245.76 MHz and called XRFdc_DynamicPLLConfig() with a ref clk equal to the PLL output clock of 3932.16 MHz.  When I fixed my external clocking to provide 3932.16 MHz to the clock input pins, I still received an error from XRFdc_MultiConverter_Sync() with return status = 20, instead of the previous return status = 2.  The output of my UART on the RFSoC board is shown below:

 

rcvBuf = MTS_Setup 0 1

txBuf = MTS_Setup
rcvBuf = MultiConverter_Init 0

txBuf = MultiConverter_Init
rcvBuf = MTS_SysRef_Config 1 0 0xF

txBuf = MTS_SysRef_Config 0 0 0 -16754944 127 -2029541816 127 800060000 0 -2029541968 127 800060000 0 -16754496 127 -16754640 127 4240732 0 0 0 4 0 1
rcvBuf = SetMixerSettings 0 0 0 0.0 0.0 3 2 0 3 0

txBuf = SetMixerSettings
rcvBuf = ResetNCOPhase 0 0 0

txBuf = ResetNCOPhase
rcvBuf = SetMixerSettings 0 0 1 0.0 0.0 3 2 0 3 0

txBuf = SetMixerSettings
rcvBuf = ResetNCOPhase 0 0 1

txBuf = ResetNCOPhase
rcvBuf = SetMixerSettings 0 1 0 0.0 0.0 3 2 0 3 0

txBuf = SetMixerSettings
rcvBuf = ResetNCOPhase 0 1 0

txBuf = ResetNCOPhase
rcvBuf = SetMixerSettings 0 1 1 0.0 0.0 3 2 0 3 0

txBuf = SetMixerSettings
rcvBuf = ResetNCOPhase 0 1 1

txBuf = ResetNCOPhase
rcvBuf = SetMixerSettings 0 2 0 0.0 0.0 3 2 0 3 0

txBuf = SetMixerSettings
rcvBuf = ResetNCOPhase 0 2 0

txBuf = ResetNCOPhase
rcvBuf = SetMixerSettings 0 2 1 0.0 0.0 3 2 0 3 0

txBuf = SetMixerSettings
rcvBuf = ResetNCOPhase 0 2 1

txBuf = ResetNCOPhase
rcvBuf = SetMixerSettings 0 3 0 0.0 0.0 3 2 0 3 0

txBuf = SetMixerSettings
rcvBuf = ResetNCOPhase 0 3 0

txBuf = ResetNCOPhase
rcvBuf = SetMixerSettings 0 3 1 0.0 0.0 3 2 0 3 0

txBuf = SetMixerSettings
rcvBuf = ResetNCOPhase 0 3 1

txBuf = ResetNCOPhase
rcvBuf = MTS_SysRef_Config 1 0 0xF

txBuf = MTS_SysRef_Config 0 0 0 -16754944 127 -2029541816 127 800060000 0 -2029541968 127 800060000 0 -16754496 127 -16754640 127 4240732 0 0 0 4 0 1
rcvBuf = MultiConverter_Sync 0 -1 15

ERROR: MultiConverter_Sync 20 -1 31 31 31 0 302 574 1030 1407 15 1 EXECUTE

*****************************

rcvBuf = GetLog

txBuf = GetLog metal: info:
DTC Scan T1
metal: info: ADC0: 3333322222000000000000000000000000001111333333333333322222000000#00000*000000000000111333333333333332222000000000000000000000000
metal: info: ADC1: 200000000000000000000000000011133333333333333222200000000000*000000000#001111333333333333333322220000000000000000000000111133333
metal: info: ADC2: 200000000000000000000000001111333333333333333222220000000000*000000000#011113333333333333332222200000000000000000000011111333333
metal: info: ADC3: 333222200000000000000000000000010111333333333333332222200000000000*000#000000011113333333333333332220000000000000000000000000111
metal: info: ADC0: Marker: - 32, 15
metal: error: Analog SysRef timeout,SysRef not detected on ADC tile 0
metal: info: ADC1: Marker: - 130, 15
metal: error: Analog SysRef timeout,SysRef not detected on ADC tile 1
metal: info: ADC2: Marker: - 187, 15
metal: error: Analog SysRef timeout,SysRef not detected on ADC tile 2
metal: info: ADC3: Marker: - 238, 15
metal: error: Analog SysRef timeout,SysRef not detected on ADC tile 3
metal: info: SysRef period in terms of ADC T1s = 512
metal: info: ADC target latency = 271
metal: info: ADC1 latency offset by a SysRef period to 543
metal: info: ADC2 latency offset by a SysRef period to 999
metal: info: ADC3 latency offset by a SysRef period to 1407
metal: error: Alignment correction delay 31 required exceeds maximum for ADC Tile 31
metal: error: Alignment correction delay 31 required exceeds maximum for ADC Tile 31
metal: error: Alignment correction delay 31 required exceeds maximum for ADC Tile 31

 

To confuse things further, I'm using a custom RFSoC board that doesn't have the same external clocking network of the ZCU111, but I'm attempting to modify the ZCU111 MTS8x8 design to run on this board.  On the custom board, there is a single LMX2582 PLL with a 122.88 MHz external VCXO.  The LMX2582 two outputs each go to Analog Devices 4-output fanout clock drivers which then drive the tile clock inputs on the RFSoCs ADCs and DACs.  For whatever reason, the board designer connected one of the four outputs of one of the clock drivers to the SYSREF input on the RFSoC, so when you try to clock the DACs at 3932.16 MHz, this frequency is going into the SYSREF input as well.

I am attempting to provide my own free-running SYSREF, generated in the FPGA fabric, for the user_sysref_adc and user_sysref_dac inputs of the RFDC IP.  For clocking, I take the adc_clk0 output of the RFDC and connect it to a BUFG which then drives the input of an MMCM with phase alignment selected.  The output clocks of the MMCM are 491.52 and 245.76 MHz and are connected via BUFGs to the m_axis_clk and s_axis_clk inputs of the RFDC IP, resepctively.  I generate my own SYSREFs of 7.68 MHz (to match the ZCU111 design) on each of the adc and dac clock domains and connect them to the user_sysref_adc and user_sysref_dac inputs of the RFDC IP.

I'm not sure what the differential SYSREF inputs for the RFDC are for if the user_sysref_adc and user_sysref_dac are supplied anyways.  Will it be impossible to use the MTS if I have a frequency of 3932.16 MHz going into the diff SYSREF input on the RFDC?

I should mention that I am confident that I'm configuring the LMX2582 correctly.  I also verify that the MMCM is locked and have clock counters that verify the frequencies of the MMCM outputs are 491.52 and 245.76 MHz.  I also have ILAs that allow me to verify my free-running SYSREFs are the exact period required.

Hope this wasn't too confusing and thanks for any help!

 

0 Kudos
Participant sasquatch
Participant
490 Views
Registered: ‎07-22-2018

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

Hello,

I enabled METAL_LOG_DEBUG instead of METAL_LOG_INFO and have a little more info on the MTS sync error.  I have a custom RFSoC board where I'm trying to clock both ADCs and DACs with an external 3.93216 GHz clock.  Unfortunately, the differential SYSREF pins of the FPGA are connected to a 3.93216 GHz clock as well, instead of some sub-multiple of this frequency.  I have generated the user_sysref_adc and user_sysref_dac signals from free-running counters in the fabric and hope to ignore the erroneous differential SYSREF pins signals.

Can anyone tell me why I'm getting the errors in the log file below?  The error is at line 89 during MultiConverter_Sync() and I retrieve the metal log right after that with GetLog().

Thanks for any help!

0 Kudos
Moderator
Moderator
425 Views
Registered: ‎04-18-2011

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

In not too sure what you mean exactly a diagram of the sysref scheme would help to clarify it. 

I don't think what you are doing is going to work. 

In this case generating your own sysref with no phase relationship to the clocks is going to cause issues 

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
Participant sasquatch
Participant
400 Views
Registered: ‎07-22-2018

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

Thanks for your reply.  I agree on the phase issue, and have recently modified my design to try to get around this.  I've attached a block diagram of the clocking on my custom RFSoC board with frequencies in red and in MHz.  If I accept that I won't be able to use the DACs, I can set the divider on RFoutB of the LMX2582 to 128, giving a SYSREF of 30.72 MHz.  The max divider setting is 3*8*8, but I'm using 2*8*8 for now.  I have added a variable delay element between my free-running SYSREF generator and the user_sysref_adc input.  The counter generates a 50% duty cycle signal at 30.72 MHz and the variable delay allows me to select between the 16 possible phases (on the 491.52 MHz clock).  After the LMX2582 locks and the MMCM locks, the phase between the external 30.72 MHz SYSREF and the internal one I am generating should be fixed.  I am using the MMCM in phase alignment mode, so that adc_clk is aligned to the clk_adc0 output of the RFDC with minimum delay.  After all the clocks are stable, I loop through 16 possible delays and send the commands for the MTS sync.  I've attached the UART output which shows the exact commands received by the RFSoC in the rcvBuf printf() statements.  After sending the MultiConverter_Sync command, I used GetLog to get the metal log output.  In some cases I see a return status of 0 from MultiConverter_Sync, but I don't think it is really succeeding.  It is not like there is a single successful MultiConverter_Sync across the 16 possible phases of user_sysref_adc and I'm not sure the DTC sweeps look correct.

Thanks for your help!

 

rfsoc_clocking.png
0 Kudos
Participant sasquatch
Participant
340 Views
Registered: ‎07-22-2018

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

I just read on p. 125 of pg269.pdf that the SYSREF period must be less than 10 MHz.  Is this a truly rigid requirement?  What sets this limit?

0 Kudos
Moderator
Moderator
314 Views
Registered: ‎04-18-2011

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

The first problem with this that you are taking the divided clock coming out of the tile and using it to create the stream clock for each tile. 

There is no guaranteed phase relationship for the tile output clocks across tiles. The best you could do with this is single chip alignment. That would be if you adhered to the other recommendations in the documentation. 

You can't just make your own PL SYSREF. 

This is not going to work reliably because you are missing the point about using a second SYSREF coming to the PL. 

The MTS procedure should use the Analog SYSREF to establish common clocking inside the tiles so everything inside the tiles is aligned. the only mismatch now is across the dual clock FIFOs.

The two SYSREFs (analog and PL) should come from the same source and they should arrive at the same time at the package balls. 

This way you have alignment between them outside.

Once this is done the PL SYSREF can be captured in the stream clock domain.

Then we use the incoming PL SYSREF to put the marker into the fifo you get a measure of the latency across the FIFO only, you don't get the fifo latency plus some external mismatch. 

The two SYSREFs in your case bear no relation to one another. How can this ever work properly in this case?

The last question about the FMAX of SYSREF. I think that we use <10Mhz so that the Software can manage the MTS without worrying about additional pulses arriving while it works. It is what we have tested so it is what we support. I think this is in line with JESD.

-------------------------------------------------------------------------
Don’t forget to reply, kudo, and accept as solution.
-------------------------------------------------------------------------
Participant sasquatch
Participant
241 Views
Registered: ‎07-22-2018

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

Thanks for your reply.  Do you think it would be possible to configure the channels without MTS, and PLLs/NCOs disabled, and have not-necessarily time-aligned samples across the channels, but at least constant latencies for each channel, assuming no reconfiguration?  If this were possible, a cal signal could be used afterwards to determine timing or phase differences across the channels.  Basically, I'd like to ignore the erroneous SYSREF signal at the package pins completely.

0 Kudos
Observer chintan_asi
Observer
74 Views
Registered: ‎11-05-2018

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

Kendrick,Keith:

If one is to use the adc_pl_event port on the RF-ADC (when enable real-time ports box is selected) for MTS, do all the PCB and clocking details of SYSREF go away? Is it as "simple" as generating a pulse N clocks wide in the axis_clock domain for the each tile? (Any criteria of N?), and the pulse will then as a common time base for all tiles instead of the sysref clock? Also, does the adc_pl_event allow equating the other components that contribute to phase non-coherency (e.g FIFO latency, etc), or is it only usable for synchronizing the NCO phase update?

Thanks

Chintan

 

0 Kudos
Observer chintan_asi
Observer
58 Views
Registered: ‎11-05-2018

Re: Procedure For Multi-Tile Synchronization of RFSoC ADCs With NCOs Enabled

@bmarcotte247
Were you able to get past your ADC MTS issue? If so, anything different you had to do from steps listed above?

Thanks
0 Kudos