Sampling Distributions and Estimation

A sample mean is itself a random variable: take another sample and you get a different value. Knowing how it varies is what makes it possible to say anything about the population from a single sample.

The sampling distribution of the mean

If a population has mean $\mu$ and variance $\sigma^2$, then for samples of size $n$:

$$E(\bar{X}) = \mu \qquad \operatorname{Var}(\bar{X}) = \frac{\sigma^2}{n}$$

The standard deviation of $\bar{X}$ is called the standard error:

$$\text{SE} = \frac{\sigma}{\sqrt{n}}$$

For $\mu = 100$, $\sigma = 15$ and $n = 25$:

$$\text{SE} = \frac{15}{5} = 3$$

So sample means cluster around 100 with a standard deviation of 3 — five times tighter than individual values.

If the population is normal, $\bar{X}$ is exactly normal:

$$\bar{X} \sim N\!\left(\mu,\ \frac{\sigma^2}{n}\right)$$

Find $P(\bar{X} > 103)$ in the example above.

$$z = \frac{103 - 100}{3} = 1 \qquad P(Z > 1) = 1 - 0.8413 = 0.1587$$

The Central Limit Theorem

For a sufficiently large sample, the distribution of $\bar{X}$ is approximately normal whatever the shape of the population.

In practice $n \ge 30$ is taken as sufficient, and less is needed when the population is already roughly symmetric.

This is what makes normal-based methods so widely applicable: the population may be skewed, discrete or oddly shaped, and the sample mean is still approximately normal.

Unbiased estimators

An estimator is unbiased if its expected value equals the parameter it estimates.

$$E(\bar{X}) = \mu \qquad \text{so } \bar{x} \text{ is an unbiased estimator of } \mu$$

For the variance the natural formula is biased — it systematically underestimates, because the deviations are measured from the sample's own mean rather than the true one. Dividing by $n-1$ corrects it:

$$s^2 = \frac{\sum(x - \bar{x})^2}{n-1} = \frac{1}{n-1}\left(\sum x^2 - \frac{(\sum x)^2}{n}\right)$$

Confidence intervals

A confidence interval gives a range of plausible values for $\mu$:

$$\bar{x} \pm z\,\frac{\sigma}{\sqrt{n}}$$

A sample of 64 has mean 50, from a population with $\sigma = 8$. Find a 95% confidence interval for $\mu$.

$$\text{SE} = \frac{8}{\sqrt{64}} = 1$$ $$50 \pm 1.96(1) = (48.04,\ 51.96)$$

At 99% confidence the same data gives

$$50 \pm 2.576(1) = (47.42,\ 52.58)$$

— a wider interval, because more confidence demands more room.

When $\sigma$ is unknown and the sample is small, $s$ replaces $\sigma$ and the $t$-distribution replaces the normal — the subject of the next topic.

Interpreting a confidence interval

The correct statement is:

If many samples were taken and an interval constructed from each in this way, about 95% of those intervals would contain the true $\mu$.

The incorrect statement, and the one most often written:

There is a 95% probability that $\mu$ lies in this interval.

$\mu$ is a fixed number, not a random one. Either it is in this particular interval or it is not; the probability refers to the procedure, not to this single result.

Choosing a sample size

The margin of error is the half-width:

$$E = z\,\frac{\sigma}{\sqrt{n}} \quad \Rightarrow \quad n = \left(\frac{z\sigma}{E}\right)^2$$

How large a sample gives a margin of error of 1 at 95% confidence, with $\sigma = 8$?

$$n = \left(\frac{1.96 \times 8}{1}\right)^2 = (15.68)^2 = 245.9$$

Round up to $n = 246$.