| Author |
Message |
sdeepa
Guest
|
 Posted:
Tue Nov 29, 2005 1:16 am Post subject:
variance vs mean square error |
|
|
can anyone please let me know how the variance of noise affects the mea
square error? Will the mean square error value increase with value o
sample variance of noise? |
|
| Back to top |
|
 |
robert bristow-johnson
Guest
|
| Posted:
Tue Nov 29, 2005 1:16 am Post subject:
Re: variance vs mean square error |
|
|
Peter K. wrote:
| Quote: |
The difference between the variance and the mean square error can be a
little subtle to understand, but there is a difference.
Just because you generate a sequence of N samples from a zero-mean
noise source of variance V does not mean that a) the mean of the
sequence is zero or b) the mean square error of the sequence is V.
Sure, the mean will _tend_ towards zero and the mean square error will
_tend_ towards V, but the sample mean and the actual mean and the
"sample variance" (mean square error) and the actual variance will
almost never be identical.
|
Peter, do you mean the difference between the actual mean and variance
of a random variable (we're calling "error") and the estimated values
of mean and variance obtained by sampling it?
if so, then the mean of the estimation *will* tent toward the true mean
(which might be zero) but the estimated variance is biased a little and
although the biased estimate tends toward the true variance, you can
with a finite number of samples get a better estimate:
http://groups.google.com/group/comp.dsp/msg/1d89215c51200101
i figger you know this but wanted to put it out there.
--
r b-j rbj@audioimagination.com
"Imagination is more important than knowledge." |
|
| Back to top |
|
|
Peter K.
Guest
|
| Posted:
Tue Nov 29, 2005 1:16 am Post subject:
Re: variance vs mean square error |
|
|
sdeepa wrote:
| Quote: | can anyone please let me know how the variance of noise affects the mean
square error? Will the mean square error value increase with value of
sample variance of noise?
|
Generally, as the noise variance increases the mean square error will
increase --- though the relationship may not be linear.
Your question is still a little ambiguous to answer and more
specifically.
The difference between the variance and the mean square error can be a
little subtle to understand, but there is a difference.
Just because you generate a sequence of N samples from a zero-mean
noise source of variance V does not mean that a) the mean of the
sequence is zero or b) the mean square error of the sequence is V.
Sure, the mean will _tend_ towards zero and the mean square error will
_tend_ towards V, but the sample mean and the actual mean and the
"sample variance" (mean square error) and the actual variance will
almost never be identical.
HTH.
Ciao,
Peter K. |
|
| Back to top |
|
|
John
Guest
|
| Posted:
Tue Nov 29, 2005 1:16 am Post subject:
Re: variance vs mean square error |
|
|
the mean square error _is_ - as far as I know - the variance of the error...
-------------
| Quote: | can anyone please let me know how the variance of noise affects the mean
square error? Will the mean square error value increase with value of
sample variance of noise? |
|
|
| Back to top |
|
|
naebad
Guest
|
| Posted:
Tue Nov 29, 2005 1:17 am Post subject:
Re: variance vs mean square error |
|
|
Yes - it is also the average power of the error.
The standard deviation (we assume zero-mean) is the rms value.
Naebad |
|
| Back to top |
|
|
Peter K.
Guest
|
| Posted:
Tue Nov 29, 2005 7:23 am Post subject:
Re: variance vs mean square error |
|
|
"robert bristow-johnson" <rbj@audioimagination.com> writes:
robert bristow-johnson wrote:
| Quote: | Peter, do you mean the difference between the actual mean and variance
of a random variable (we're calling "error") and the estimated values
of mean and variance obtained by sampling it?
if so, then the mean of the estimation *will* tent toward the true mean
(which might be zero) but the estimated variance is biased a little and
although the biased estimate tends toward the true variance, you can
with a finite number of samples get a better estimate:
http://groups.google.com/group/comp.dsp/msg/1d89215c51200101
i figger you know this but wanted to put it out there.
|
Ya. That's the estimate I use; any other is biased. It's good to see
a derivation of why to use it. :-)
Ciao,
Peter K. |
|
| Back to top |
|
|
Peter K.
Guest
|
| Posted:
Tue Nov 29, 2005 7:27 am Post subject:
Re: variance vs mean square error |
|
|
"robert bristow-johnson" <rbj@audioimagination.com> writes:
| Quote: | Peter, do you mean the difference between the actual mean and variance
of a random variable (we're calling "error") and the estimated values
of mean and variance obtained by sampling it?
|
Not really. I just mean that if you plug mu and sigma^2 into a random
number generator, then you shouldn't expect the estimates of the mean
and variance of the resulting numbers to be precisely mu and sigma^2
--- regardless of what estimator you use.
| Quote: | if so, then the mean of the estimation *will* tent toward the true mean
(which might be zero) but the estimated variance is biased a little and
although the biased estimate tends toward the true variance, you can
with a finite number of samples get a better estimate:
http://groups.google.com/group/comp.dsp/msg/1d89215c51200101
i figger you know this but wanted to put it out there.
|
OK, I suppose I always use the unbiased variance estimator, so I don't
tend to think about the 1/N version. Good to see _why_ to do it posted
somewhere, though, thanks for the link.
Ciao,
Peter K. |
|
| Back to top |
|
|
Jani Huhtanen
Guest
|
| Posted:
Tue Nov 29, 2005 9:16 am Post subject:
Re: variance vs mean square error |
|
|
John wrote:
| Quote: | the mean square error _is_ - as far as I know - the variance of the
error...
|
Actually mean square error is sample variance which in turn is just an
estimator for the true variance. Sample variance is "accurate" if the noise
is white noise, but generally one should take the probability distribution
into account.
http://mathworld.wolfram.com/Variance.html
--
Jani Huhtanen
Tampere University of Technology, Pori |
|
| Back to top |
|
|
Peter K.
Guest
|
| Posted:
Tue Nov 29, 2005 5:17 pm Post subject:
Re: variance vs mean square error |
|
|
Jani Huhtanen wrote:
| Quote: | Actually mean square error is sample variance
|
Only if the relationship between where the noise is added and where
you're calculating the mean square error is direct:
y = x + n
so you're calculating the mean square error of y from x and you're
talking about the sample variance of y (about mean x).
More generally, the mean square error is taken between x and xhat:
y = f(x) + n
xhat = fhat(y)
in which case the mean square error is almost completely different from
the sample variance.
| Quote: | which in turn is just an estimator for the true variance.
|
Ayup.
| Quote: | Sample variance is "accurate" if the noise
is white noise, but generally one should take the probability distribution
into account.
|
You mean there's a variance on the variance estimator? Never! ;-)
Ciao,
Peter K. |
|
| Back to top |
|
|
robert bristow-johnson
Guest
|
| Posted:
Wed Nov 30, 2005 1:16 am Post subject:
Re: variance vs mean square error |
|
|
Jani Huhtanen wrote:
| Quote: | John wrote:
the mean square error _is_ - as far as I know - the variance of the
error...
Actually mean square error is sample variance which in turn is just an
estimator for the true variance. Sample variance is "accurate" if the noise
is white noise, but generally one should take the probability distribution
into account.
|
it's a semantic issue i guess, but even given the meaning of "sample
variance" taken from the Wolfram link, i am not sure what you mean by
"accurate", Jani. the sample variance (divide by N) is less accurate
than the unbiased estimator (divide by N-1 instead of N) for getting to
the "population variance" which is the variance in the p.d.f. this is
true whether tha r.v. is white or not.
r b-j |
|
| Back to top |
|
|
robert bristow-johnson
Guest
|
| Posted:
Wed Nov 30, 2005 9:16 am Post subject:
Re: variance vs mean square error |
|
|
Jani Huhtanen wrote:
| Quote: |
By "accurate" I meant that sample variance is close to the population
variance when P(x_i) is equal for all x_i. Only that we don't know the
sample mean.
|
well, you can compute it. and the expectation value for the sample
mean *is* the population mean.
| Quote: | Assuming we know that we have zero mean noise with equal
P(x_i) for all x_i then sample variance is population variance.
|
let's say you have a regular old decent random variable with a non-zero
variance. now suppose you sample it exactly *once*.
sample mean is whatever your sample ends up being. but the expectation
value for the mean *is* the population mean (the probabilistic mean of
the r.v.).
the sample variance is always zero (for a single sample). the
expectation value for it is zero, but the population variance (the
probabilistic variance of the r.v.) is not zero.
the expectation of the sample variance is (N-1)/N times the population
variance, where N is the number of samples. they're slightly
different.
| Quote: | Then again I might be wrong...
|
maybe. again check the thread at:
http://groups.google.com/group/comp.dsp/browse_frm/thread/330ac90a92f8dfaf/1d89215c51200101
the expectation value for the sample variance is *not* (precisely) the
population variance.
r b-j |
|
| Back to top |
|
|
Jani Huhtanen
Guest
|
| Posted:
Wed Nov 30, 2005 9:16 am Post subject:
Re: variance vs mean square error |
|
|
robert bristow-johnson wrote:
| Quote: |
Jani Huhtanen wrote:
John wrote:
the mean square error _is_ - as far as I know - the variance of the
error...
Actually mean square error is sample variance which in turn is just an
estimator for the true variance. Sample variance is "accurate" if the
noise is white noise, but generally one should take the probability
distribution into account.
it's a semantic issue i guess, but even given the meaning of "sample
variance" taken from the Wolfram link, i am not sure what you mean by
"accurate", Jani.
|
By "accurate" I meant that sample variance is close to the population
variance when P(x_i) is equal for all x_i. Only that we don't know the
sample mean. Assuming we know that we have zero mean noise with equal
P(x_i) for all x_i then sample variance is population variance. Then again
I might be wrong...
--
Jani Huhtanen
Tampere University of Technology, Pori |
|
| Back to top |
|
|
Jani Huhtanen
Guest
|
| Posted:
Wed Nov 30, 2005 4:42 pm Post subject:
Re: variance vs mean square error |
|
|
robert bristow-johnson wrote:
| Quote: | Jani Huhtanen wrote:
By "accurate" I meant that sample variance is close to the population
variance when P(x_i) is equal for all x_i. Only that we don't know the
sample mean.
well, you can compute it. and the expectation value for the sample
mean *is* the population mean.
|
This just shows that I'm not really good with statistics. I though we don't
generally know the expectation value. When you calculate the sample mean
from N zero mean random samples, you don't usually get exactly zero mean do
you?
I will.
| Quote: |
the expectation value for the sample variance is *not* (precisely) the
population variance.
|
Perhaps I leave this conversation as clearly I'm not really good with
statistics ;)
--
Jani Huhtanen
Tampere University of Technology, Pori |
|
| Back to top |
|
|
Stan Pawlukiewicz
Guest
|
| Posted:
Wed Nov 30, 2005 5:17 pm Post subject:
Re: variance vs mean square error |
|
|
sdeepa wrote:
| Quote: | can anyone please let me know how the variance of noise affects the mean
square error? Will the mean square error value increase with value of
sample variance of noise?
I'm not sure I understand what your asking, so this a guess on my part. |
Usually more noise gives you a higher mean square error. In most cases,
but not all, as the noise goes to infinity, the error goes to infinity.
For a MAP estimator the error will likely be bounded by the prior.
The precise behavior of the estimate depends on the actual estimator
used. This is usually a hard problem to do analytically. This is also
why books on estimation cover topics like the Cramer Rao bound which are
relatively easy to compute. Since your asking about mean square error, I
think you want to know about the MAP estimator. There are modifications
to the CRB for priors and also for bias. |
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in