| Author |
Message |
spasmous
Guest
|
 Posted:
Mon Dec 19, 2005 1:15 am Post subject:
Median filtering and FFT |
|
|
I am making repeated measurements of a transient response to an
excitation. To improve SNR I repeat the measurement several times and
then average the results. However there are occasional "problem"
measurements that are due to unknown glitches that I don't care to
track down right now :) I'll just call them outliers.
To suppress the outliers I want to use median.
The signal I record f(t) consists of 1000 points and takes about 10ms.
I repeat this measurement every 1s or so. With averaging I just
accumlutate the measurements in a vector and FFT at the end. With
median, I'm not so sure; specifically, I can't decide which of the
following to do:
(1) Store all the measurements separately, median filter, then FFT.
(2) Store all the measurements separately, FFT, then median filter.
The computation time is not important, just the outlier suppression.
Any comments? Thanks. |
|
| Back to top |
|
 |
John Herman
Guest
|
| Posted:
Mon Dec 19, 2005 1:15 am Post subject:
Re: Median filtering and FFT |
|
|
Use 2.
But a better choice is to discard measurements that have glitches. This is
called censoring.
In article <1134935630.511232.157070@g44g2000cwa.googlegroups.com>, "spasmous"
<spasmous@gmail.com> wrote:
| Quote: | I am making repeated measurements of a transient response to an
excitation. To improve SNR I repeat the measurement several times and
then average the results. However there are occasional "problem"
measurements that are due to unknown glitches that I don't care to
track down right now :) I'll just call them outliers.
To suppress the outliers I want to use median.
The signal I record f(t) consists of 1000 points and takes about 10ms.
I repeat this measurement every 1s or so. With averaging I just
accumlutate the measurements in a vector and FFT at the end. With
median, I'm not so sure; specifically, I can't decide which of the
following to do:
(1) Store all the measurements separately, median filter, then FFT.
(2) Store all the measurements separately, FFT, then median filter.
The computation time is not important, just the outlier suppression.
Any comments? Thanks.
|
|
|
| Back to top |
|
|
Fred Marshall
Guest
|
| Posted:
Mon Dec 19, 2005 1:15 am Post subject:
Re: Median filtering and FFT |
|
|
"spasmous" <spasmous@gmail.com> wrote in message
news:1134935630.511232.157070@g44g2000cwa.googlegroups.com...
| Quote: | I am making repeated measurements of a transient response to an
excitation. To improve SNR I repeat the measurement several times and
then average the results. However there are occasional "problem"
measurements that are due to unknown glitches that I don't care to
track down right now :) I'll just call them outliers.
To suppress the outliers I want to use median.
The signal I record f(t) consists of 1000 points and takes about 10ms.
I repeat this measurement every 1s or so. With averaging I just
accumlutate the measurements in a vector and FFT at the end. With
median, I'm not so sure; specifically, I can't decide which of the
following to do:
(1) Store all the measurements separately, median filter, then FFT.
(2) Store all the measurements separately, FFT, then median filter.
|
A median filter seems an unusual thing to do on 1-D data. And, it's
nonlinear - which for your purposes may be fine.
(1) above seems much more likely. Otherwise, (2) will take into
consideration the outliers and spread the information across the result of
the FFT. Median filtering thereafter will deal with something entirely
different: spectral lines in the data or modes in the system response. I'm
sure that's *not* what you want.
In a sense, median filtering is a combination of censoring (or throwing out)
the outliers and replacing them with something more sensible. At least you
keep the sample rate. Otherwise you have to deal with irregular sample
interval issues. Throwing a sample out isn't the same as zeroing it - so
you have to decide what to do about that.
Fred |
|
| Back to top |
|
|
Joerg
Guest
|
| Posted:
Mon Dec 19, 2005 8:22 am Post subject:
Re: Median filtering and FFT |
|
|
Hello,
| Quote: | ... However there are occasional "problem"
measurements that are due to unknown glitches that I don't care to
track down right now :) I'll just call them outliers.
|
Are these outliers spikes after an AD conversion? If you cannot cure
them via EMI mitigation maybe you could do a transient analysis, like if
the data jumps by more than x that data word gets discarded and replaced
by the value of the previous word. Or by the average of the two
neighbors. It's usually best to clean this stuff out at the younger
stages of a signal processing chain.
Regards, Joerg
http://www.analogconsultants.com |
|
| Back to top |
|
|
Ulrich Bangert
Guest
|
| Posted:
Mon Dec 19, 2005 9:15 am Post subject:
Re: Median filtering and FFT |
|
|
You are right in saying that the median is insensitive to outliers. However,
the median is a single sampled value and has the same noise properties as
your other samples. You may call it the sample that is closest to outlier
insensitive average of your data. Since noise reduction seems to be part of
what you plan, i suggest you look up the so called IQR (inter quartile
range). This will be close to what you need. In general, google for "robust
statistics" for problems like that. Be prepared that computing time may
indeed become a issue with robust statistics!
Regards
Ulrich Bangert
"spasmous" <spasmous@gmail.com> schrieb im Newsbeitrag
news:1134935630.511232.157070@g44g2000cwa.googlegroups.com...
| Quote: | I am making repeated measurements of a transient response to an
excitation. To improve SNR I repeat the measurement several times and
then average the results. However there are occasional "problem"
measurements that are due to unknown glitches that I don't care to
track down right now :) I'll just call them outliers.
To suppress the outliers I want to use median.
The signal I record f(t) consists of 1000 points and takes about 10ms.
I repeat this measurement every 1s or so. With averaging I just
accumlutate the measurements in a vector and FFT at the end. With
median, I'm not so sure; specifically, I can't decide which of the
following to do:
(1) Store all the measurements separately, median filter, then FFT.
(2) Store all the measurements separately, FFT, then median filter.
The computation time is not important, just the outlier suppression.
Any comments? Thanks.
|
|
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in