Estimator for Mean and Variance of Sampled Data

Machine Learning, Science & Engineering 4 Minutes

Estimator is a statistic, usually in a function of the data, that is used to infer the value of an unknown parameter in a statistical model. During this post, we will talk about estimator for mean and variance of sampled data. We can determine a good estimator by calculating the bias of it. A good estimator should give bias closed to zero. Let \theta is parameter we want to estimate/observe, our estimator result will be \hat{\theta}. The bias of our estimator is defined as follows.

bias = E[\hat{\theta}]-\theta

We will use bias formula above to check whether our estimator is good or not. And during this post, we will check our estimator we already derived by MLE here, which are mean and variance. Let’s write them first.

\mu_{MLE}=\hat{\mu}=\frac{1}{n}\sum_{i=1}^{n}x_i\\\\ \sigma^2_{MLE}=\hat{\sigma}^2=\frac{1}{n}\sum_{i=1}^{n}(x_i-\hat{x})^2

where \bar{x}=\mu and \hat{x}=\hat{\mu}

Let’s do \mu_{MLE} first. Here we go.

E[\hat{\mu}]=E[\frac{1}{n}\sum_{i=1}^{n}x_i]\\\\ E[\hat{\mu}]=\frac{1}{n}\sum_{i=1}^{n}E[x_i]\\\\ E[\hat{\mu}]=\frac{1}{n}\sum_{i=1}^{n}\bar{x}\\\\ E[\hat{\mu}]=\frac{1}{n}n\bar{x}\\\\ E[\hat{\mu}]=\bar{x}

For our estimator \hat{\mu} above, bias = E[\hat{\mu}]-\mu=\bar{x}-\bar{x}=0. Thus, \hat{\mu}=\frac{1}{n}\sum_{i=1}^{n}x_i is a good estimator for mean \mu, since it gives us zero bias.

Let’s do for variance.

E[\sigma^2_{MLE}]=E[\frac{1}{n}\sum_{i=1}^{n}(x_i-\hat{\mu})^2]\\\\ E[\sigma^2_{MLE}]=E[\frac{1}{n}\sum_{i=1}^{n}((x_i-\mu)-(\hat{\mu}-\mu))^2]

To make it easy to derive, let (x_i-\mu)=a and (\hat{\mu}-\mu)=b. Thus, our last equation becomes:

E[\hat{\sigma}^2]=E[\frac{1}{n}\sum_{i=1}^{n}(x_i-\hat{\mu})^2]\\\\ E[\hat{\sigma}^2]=E[\frac{1}{n}\sum_{i=1}^{n}((x_i-\mu)-(\hat{\mu}-\mu))^2]\\\\ E[\hat{\sigma}^2]=E[\frac{1}{n}\sum_{i=1}^{n}(a-b)^2]\\\\ E[\hat{\sigma}^2]=E[\frac{1}{n}\sum_{i=1}^{n}(a^2-2ab+b^2)]\\\\ E[\hat{\sigma}^2]=E[\frac{1}{n}\sum_{i=1}^{n}a^2]-E[\frac{2}{n}\sum_{i=1}^{n}(ab)]+E[\frac{1}{n}\sum_{i=1}^{n}(b^2)]\\\\ E[\hat{\sigma}^2]=E[\frac{1}{n}\sum_{i=1}^{n}a^2]-E[\frac{2}{n}b\sum_{i=1}^{n}(a)]+E[\frac{1}{n}n(b^2)]\,... \,(i)\\\\ E[\hat{\sigma}^2]=E[\frac{1}{n}\sum_{i=1}^{n}a^2]-E[\frac{2}{n}bnb]+E[b^2]\,... \,(ii)\\\\ E[\hat{\sigma}^2]=E[\frac{1}{n}\sum_{i=1}^{n}a^2]-2E[b^2]+E[b^2]\\\\ E[\hat{\sigma}^2]=E[\frac{1}{n}\sum_{i=1}^{n}a^2]-E[b^2]\\\\ E[\hat{\sigma}^2]=E[\frac{1}{n}\sum_{i=1}^{n}(x_i-\mu)^2]-E[(\hat{\mu}-\mu)^2]\\\\ E[\hat{\sigma}^2]=\sigma^2-E[(\hat{\mu}-E[\hat{\mu}])^2]\,... \,(iii)\\\\ E[\hat{\sigma}^2]=\sigma^2-\frac{\sigma^2}{n}\,... \,(iv)

Let’s us clarify first how we can get (i), (ii), (iii). (i) because b is just a constant since \hat{\mu} and {\mu} are constant. (ii) because \sum_{i=}^{n}a=nb. Here is the detail.

\sum_{i=1}^{n}a=\sum_{i=1}^{n}(x_i-\mu)\\\\ \sum_{i=1}^{n}a=\sum_{i=1}^{n}(x_i)-\sum_{i=1}^{n}(\mu)\\\\ \sum_{i=1}^{n}a=n\frac{\sum_{i=1}^{n}(x_i)}{n}-n\mu\\\\ \sum_{i=1}^{n}a=n\hat{\mu}-n\mu=n(\hat{\mu}-\mu)=nb

(iii), it’s because E[\hat{\mu}]=\hat{x}=\mu. And as for (iv), here is the detail.

E[(\hat{\mu}-E[\hat{\mu}])^2]=var[\hat{\mu}]\\\\ var[\hat{\mu}]=var[\frac{\sum_{i=1}^{n}x_i}{n}]\\\\ var[\hat{\mu}]=\frac{1}{n^2}[var(x_1)+var(x_2)+... +var(x_n)]\\\\ var[\hat{\mu}]=\frac{1}{n^2} \,n\,\sigma^2\\\\ var[\hat{\mu}]=\frac{\sigma^2}{n}

And now let’s check the bias of this estimator.

bias=E[\hat{\sigma}^2]-sigma^2=(\sigma^2-\frac{\sigma^2}{n}))-\sigma^2\\\\bias=-\frac{\sigma^2}{n}

From equation above we see that variance we get from MLE does not give us zero bias. That’s why when we calculate variance of sampled data, we usually use this formula \rightarrow \hat{\sigma}^2=\frac{1}{(n-1)}\sum_{i=1}^{n}(x_i-\hat{\mu})^2. Because this formula will give us zero bias. Let’s prove it.

E[\hat{\sigma}^2]=E[\frac{1}{(n-1)}\sum_{i=1}^{n}(x_i-\hat{\mu})

Doing similar with what we already did before, we will get:

E[\hat{\sigma}^2]=E[\frac{1}{(n-1)}\sum_{i=1}^{n}a]-E[\frac{n}{n-1}b^2]\\\\ E[\hat{\sigma}^2]=E[\frac{n}{(n-1)}\frac{1}{n}\sum_{i=1}^{n}a]-E[\frac{n}{n-1}b^2]\\\\ E[\hat{\sigma}^2]=\frac{n}{(n-1)}\sigma^2-\frac{n}{n-1}E[b^2]\\\\ E[\hat{\sigma}^2]=\frac{n}{(n-1)}\sigma^2-\frac{n}{n-1}\frac{\sigma^2}{n}\\\\ E[\hat{\sigma}^2]=\frac{n}{(n-1)}\sigma^2-\frac{1}{n-1}\sigma^2\\\\ E[\hat{\sigma}^2]=(\frac{n}{(n-1)}-\frac{1}{n-1})\sigma^2\\\\ E[\hat{\sigma}^2]=(\frac{n-1}{(n-1)})\sigma^2\\\\ E[\hat{\sigma}^2]=(1)\sigma^2\\\\ E[\hat{\sigma}^2]=\sigma^2

And now let’s check the bias of this estimator.

bias=E[\hat{\sigma}^2]-sigma^2=\sigma^2-\sigma^2\\\\bias=0

Voila! We can successfully derive that by using  \hat{\sigma}^2=\frac{1}{(n-1)}\sum_{i=1}^{n}(x_i-\hat{\mu})^2  as our variance estimator for sampled data, it will give us zero bias.

Published by ardianumam

Published

Leave a comment