topic Re: Frequency estimation of a decaying sinusoidal wave with exponential background in AI Engine, DSP IP and Tools
https://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170834#M4814
<P>It's not clear how much noise you have. The data you attached to your first post did not have</P>
<P>any clearly visible noise in it. This could affect how you need to proceed. Obviously enough</P>
<P>noise can add zero-crossings not related to the sinusoidal signal. It is not very expensive</P>
<P>in terms of FPGA resources to filter the data (low pass or band pass) before applying the</P>
<P>zero crossing measurement. This is mathematically similar to using more points to interpolate</P>
<P>the point of zero crossing. Filtering is commonly applied for things like video data before</P>
<P>doing data interpolation for up or downsampling.</P>
<P> </P>
<P>-- Gabor</P>Wed, 10 Aug 2011 19:27:46 GMTgszakacs2011-08-10T19:27:46ZFrequency estimation of a decaying sinusoidal wave with exponential background
https://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170770#M4810
<P>Hello guys :),</P>
<P> </P>
<P>I have a device that produces a signal every 20 ms like the one in the attached file. The data in the file is produced by a simulation of the experiment with pink (1/f) and white noise added to the signal. The frequency is 3.5 kHz, sampling rate is 10^6/s, number of points 20000 (or less after cropping the sides which has non-useful data). My task is to estimate the frequency of the signal, and in the future this has to be implemented in an FPGA (with VHDL). The signal can be written in the following form in General:</P>
<P> </P>
<P>S(t) = (A - B Exp[-F t]) + a Exp[-b t] Sin[2 Pi w t + phi],</P>
<P> </P>
<P>where A,B,F,a,b,w,phi are fitting parameters.</P>
<P> </P>
<P>I could do very accurate fitting with this form get the frequencies very close to the real value (3.5 kHz), and the Allan standard deviation was very good for the signal.</P>
<P> </P>
<P>I'm a physicist, that's why I'm not so familiar with signal processing :-)</P>
<P> </P>
<P>----------</P>
<P>The problem:</P>
<P> </P>
<P>The problem now is that I find fitting a very complicated process for an FPGA. I want to find a simpler way to do the fit.</P>
<P>Using list convolving with a constant kernal (whose length is equal to the a single period in the signal. Is there a better way?) I could subtract the background exponential to a very good level, so I'm left with a sinusoidal wave in an exponential decaying envelope and some pink and white noise with it. </P>
<P> </P>
<P>I'm not sure what I could do to estimate the frequency of this wave. Applying Fourier transforms directly is not helping, because the period in which the signal was recorded is 20 ms, which means that the bandwidth resolution is approximately 50 Hz depending on the number of points I crop from the front and end of the signal. I could improve the result using Fourier transform to plus/minus 2 Hz of the real value by making the beginning and end of the signal in-phase by cropping some more points from the beginning and end of the signal.</P>
<P> </P>
<P>I still need more accuracy!</P>
<P> </P>
<P>For this I thought of many things. Could you guys tell me how good you see them as professionals, and how possible it's to apply them to an FPGA?</P>
<P> </P>
<P>1- The signal can be resampled, in a way that makes the first and last point exactly match in phase, which will give a even higer accuracy in the peak produced by the Forier transform.</P>
<P> </P>
<P>2- A linear fit can be done at every intersection to the y=0 axis, in a way that calculates the point that hits the axis and by counting the points in a period we can estimate the frequency.</P>
<P> </P>
<P>Any other ideas are very highly appreciated :-)</P>
<P> </P>
<P>Thanks a lot!</P>Wed, 10 Aug 2011 16:25:39 GMThttps://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170770#M4810thedestroyer2011-08-10T16:25:39ZRe: Frequency estimation of a decaying sinusoidal wave with exponential background
https://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170772#M4811
<P>t.d.</P>
<P> </P>
<P>What about remainung in the time-domain? One can find the time between zer-crossings to a better degree of accuracy, perhaps sampling at 400 MHz (+/- 2.5 ns).</P>
<P> </P>
<P>A signature of zero-crossings could be recognized.</P>
<P> </P>
<P> </P>Wed, 10 Aug 2011 16:31:52 GMThttps://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170772#M4811austin2011-08-10T16:31:52ZRe: Frequency estimation of a decaying sinusoidal wave with exponential background
https://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170778#M4812
<DIV style="color: #000000; font-family: Arial; font-size: 12px; background-image: none; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: #ffffff; line-height: 15px; background-repeat: no-repeat no-repeat; padding: 0px; margin: 5px;">
<P>Thank you for your answer! :-)</P>
<P> </P>
<P>Sampling with a higher rate is, unfortunately, not possible for technical reasons that have to do with the measurement device we're using. The zero-crossing thing is my second proposal in the solutions I propesed (sorry for not using the right term). Using only 2 points to obtain the zero-crossing is not a good idea since there is noise. So I thought of using a linear fit to obtain the zero-crossings.</P>
<P> </P>
<P>Is this practically possible? I think fitting a line is a very expensive process for an FPGA. Isn't it?</P>
<P> </P>
</DIV>Wed, 10 Aug 2011 16:43:52 GMThttps://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170778#M4812thedestroyer2011-08-10T16:43:52ZRe: Frequency estimation of a decaying sinusoidal wave with exponential background
https://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170814#M4813
Why is sampling with with a higher rate not possible? There is always a 100 MHz crystal oscillator out there somewhere on just about every board sold.<BR /><BR />Hysteresis should be used on the input to reject noise for the zero-crossings.<BR /><BR />You would have 20 ms of zero-crossings. If the nominal frequency is 3.5 KHz, that is about 70 cycles, or 140 zero-crossings. That is all you get. But you will know each zero-crossing +/- 10ns for a 100 MHz sample rate.<BR />Wed, 10 Aug 2011 18:02:03 GMThttps://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170814#M4813austin2011-08-10T18:02:03ZRe: Frequency estimation of a decaying sinusoidal wave with exponential background
https://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170834#M4814
<P>It's not clear how much noise you have. The data you attached to your first post did not have</P>
<P>any clearly visible noise in it. This could affect how you need to proceed. Obviously enough</P>
<P>noise can add zero-crossings not related to the sinusoidal signal. It is not very expensive</P>
<P>in terms of FPGA resources to filter the data (low pass or band pass) before applying the</P>
<P>zero crossing measurement. This is mathematically similar to using more points to interpolate</P>
<P>the point of zero crossing. Filtering is commonly applied for things like video data before</P>
<P>doing data interpolation for up or downsampling.</P>
<P> </P>
<P>-- Gabor</P>Wed, 10 Aug 2011 19:27:46 GMThttps://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170834#M4814gszakacs2011-08-10T19:27:46ZRe: Frequency estimation of a decaying sinusoidal wave with exponential background
https://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170908#M4815
<P>Thank you for your answer.</P>
<P> </P>
<P>It's not possible to sample in a higher rate because there are other processes involved and it's more complicated than it seems. The physics of the experiment doesn't allow that.</P>
<P> </P>
<P>Could you please answer my question about whether it's possible to make a linear fit (regression) in an FPGA? I'm gonna make a simulation for linear fitting today to see how accurate it's to use a linear fit at every zero crossing to get the frequency.</P>
<P> </P>
<P>And are there any other methods to estimate the frequency?</P>Thu, 11 Aug 2011 06:42:57 GMThttps://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170908#M4815thedestroyer2011-08-11T06:42:57ZRe: Frequency estimation of a decaying sinusoidal wave with exponential background
https://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170910#M4816
<P>Thank you for your reply.</P>
<P> </P>
<P>I provided a sample of the data in a text file with in the attachment with the picture. If you fit the data to the equation I provided, the residuals will represent the pink and white noise I added.</P>
<P> </P>
<P>And why would I want to use filters? please explain!</P>Thu, 11 Aug 2011 06:46:56 GMThttps://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170910#M4816thedestroyer2011-08-11T06:46:56ZRe: Frequency estimation of a decaying sinusoidal wave with exponential background
https://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170978#M4817
<BLOCKQUOTE><HR /><LI-USER uid="24958"></LI-USER> wrote:<BR />
<P>Thank you for your reply.</P>
<P> </P>
<P>I provided a sample of the data in a text file with in the attachment with the picture. If you fit the data to the equation I provided, the residuals will represent the pink and white noise I added.</P>
<P> </P>
<P>And why would I want to use filters? please explain!</P>
<HR /></BLOCKQUOTE>
<P>In that case it looks like the pink and white noise are quite small compared to the signal. It wasn't</P>
<P>clear from looking at the waveform that there was any noise added. The real question then comes</P>
<P>down to whether you are trying to measure the frequency with more accuracy than cam be obtained</P>
<P>from the zero-crossings with the noise included. If not, then you don't really need filtering. If you</P>
<P>do, then a bandpass filter could remove a significant portion of the noise.</P>
<P> </P>
<P>Another effect of filters is to remove "shot noise" or more simply put single samples that don't</P>
<P>fit the waveform. If you were to choose a linear interpolation approach to finding the zero-</P>
<P>crossings, then the filtered waveform would be a better choice. The filtered value of the</P>
<P>points just before and after the crossing would already be affected by the surrounding</P>
<P>points, making the zero-crossing computation similar to fitting a line to more data points.</P>
<P> </P>
<P>What's not as clear to me is how you define a "zero crossing" when the center of the sinusoid</P>
<P>changes over time. Obviously the fitting approach takes this into account. For a more simplistic</P>
<P>time-domain approach you may need to continually monitor the wave peaks and compute the</P>
<P>"zero" level. Note that a bandpass filter would also remove some of the background offset.</P>
<P> </P>
<P>-- Gabor</P>Thu, 11 Aug 2011 13:57:20 GMThttps://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/170978#M4817gszakacs2011-08-11T13:57:20ZRe: Frequency estimation of a decaying sinusoidal wave with exponential background
https://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/171030#M4818
<P>Thank you again for your reply.</P>
<P> </P>
<P>About the center of the sine changing, I convolved the signal with a constant kernal whose length is equal to the signal's average frequency (10^6/3500 = 286 samples).</P>
<P> </P>
<P>Yes! I need an extremely high accuracy, I'm looking for an error +/- 10^-10 Hz!</P>
<P> </P>
<P>I created today the script that does the linear fitting. The script removes background shift, and then detects the points that are different in sign, then creates a linear fit of the 2N points around them (on both sides). The result from this approach weren't very pleasing. The results are the following in a list for 100 decays (100 of the signals like the one I posted in my first post) :</P>
<P> </P>
<P>{3500.07,3500.24,3500.02,3500.22,3500.17,3500.27,3500.08,3499.98,3499.89,3500.33,3500.3,3500.07,3500.13,3500.2,3500.09,3500.03,3500.24,3500.,3500.3,3500.19,3499.9,3500.09,3500.17,3500.27,3500.36,3499.87,3500.17,3500.22,3499.97,3499.95,3499.98,3500.06,3499.88,3500.34,3500.1,3500.03,3500.13,3500.14,3500.02,3500.03,3500</P>
<P> </P>
<P>You're right, noise has to be removed to make this better, I noticed this when I found some contiguous points having negative and positive values. I'm a physicist and I'm not so familiar with filters, so I'm gonna start learning filters in some more details (I'm only familiar with simple filters like low-pass). But someone told me that using a low-pass filter or a high-pass filter or a band-pass filter may have an effect on the frequency on a phase dependent way, which will complicate things way out of my league! is this true?</P>
<P> </P>
<P>could you please tell me how to start with this?</P>Thu, 11 Aug 2011 15:25:47 GMThttps://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/171030#M4818thedestroyer2011-08-11T15:25:47ZRe: Frequency estimation of a decaying sinusoidal wave with exponential background
https://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/171248#M4819
<P><FONT color="#FF0000">Yes! I need an extremely high accuracy, I'm looking for an error +/- 10^-10 Hz!</FONT></P>
<P></P>
<P> </P>
<P>Ouch! I don't think it very likely that you'll get that sort of accuracy with a simple</P>
<P>zero-crossing method. Your original curve-fitting approach essentially used</P>
<P>all of the input data points to generate the best fit. Your line fitting approach can</P>
<P>only use a small subset of the data where the trajectory is relatively linear.</P>
<P>You might still want to try using a fairly long digital bandpass filter in front of</P>
<P>the zero-crossing technique. As you mentioned, this does affect the phase</P>
<P>of the signal, but as I understand you're only interested in the frequency,</P>
<P>which would not be affected by the filter. There could be some secondary effects</P>
<P>of the amplitude modulation that affect the frequency measurement after</P>
<P>filtering, but theoretically you can compensate for this if the incoming</P>
<P>wave shape is constant. The important thing is to find a method with the</P>
<P>required precision. Then you can find the correction factors to get the required</P>
<P>accuracy.</P>
<P> </P>
<P>-- Gabor</P>Fri, 12 Aug 2011 13:38:54 GMThttps://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/171248#M4819gszakacs2011-08-12T13:38:54ZRe: Frequency estimation of a decaying sinusoidal wave with exponential background
https://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/171276#M4820
<P>Thank you for your reply!</P>
<P> </P>
<P>WoW!!!! I used 5 low pass and 5 high pass filters, and could impressively clean the signal from noise and background shifts in a very effective way!!! I fitted the line to a sine with an exponential and the fit was perfect with 10^-12 residuals!!!! I'm impressed, thank you for the idea!</P>
<P> </P>
<P>Please find attached a sample of the filtered data. The plot is the fit with connected data points. Blue is the fit.</P>
<P> </P>
<P>The filters I used have 2000 Hz to 15000 Hz band-pass.</P>
<P> </P>
<P>I could get get with line fittings nice results with up to +/- 0.3 Hz maximum errors! So the question is now, how could I improve this? Let's say I'm looking for precision of +/- 0.001 Hz. Is that ever possible by improving the signal or by any other estimation method?</P>Fri, 12 Aug 2011 15:27:01 GMThttps://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/171276#M4820thedestroyer2011-08-12T15:27:01ZRe: Frequency estimation of a decaying sinusoidal wave with exponential background
https://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/171436#M4821
<P>How realistic is this error margin?</P>
<P> </P>
<P>If you are filtering an input signal, what is the error of the sampling or generating timebase (including noise, jitter, etc.) ?</P>
<P>If the FPGA is the signal generator, how accurate is the FPGA's timebase?</P>
<P> </P>
<P>-- Bob Elkind</P>Sat, 13 Aug 2011 18:28:16 GMThttps://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/171436#M4821eteam002011-08-13T18:28:16ZRe: Frequency estimation of a decaying sinusoidal wave with exponential background
https://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/171462#M4822
<P>Thank you for your answer!</P>
<P> </P>
<P>Using fitting with that complicated function I expressed in the first post, the error was very small, and The Allan Standard Deviation for the frequency of 4.9*10^5 measurements (decays like the ones I attached) was stable from the simulation.</P>
<P> </P>
<P>About being not accurate in the FPGA for precission reasons is not the issue right now. I want a theoretical method that I could apply on C++ or Mathematica and gain the best accuracy ever, and then lateron implement this method on an FPGA and gain errors that only have to do with precission in the FPGA.</P>
<P> </P>
<P>In the real experiment, the signal isn't generated by an FPGA, but by an ADC that gets its result from a real experiment, and then received by an FPGA for detecting the frequenecy.</P>
<P> </P>
<P>Does this make sense? Am I on the right path?</P>Sun, 14 Aug 2011 05:28:08 GMThttps://forums.xilinx.com/t5/AI-Engine-DSP-IP-and-Tools/Frequency-estimation-of-a-decaying-sinusoidal-wave-with/m-p/171462#M4822thedestroyer2011-08-14T05:28:08Z