Hypothesis Testing

A hypothesis test asks whether a sample result is surprising enough to reject a stated claim about the population. Every test follows the same five steps; only the test statistic changes.

Hypotheses

The null hypothesis $H_0$ states the value being tested — the status quo, the manufacturer's claim, "no change".

The alternative hypothesis $H_1$ states what you suspect instead.

$$H_0: \mu = 500 \qquad H_1: \mu \ne 500$$

A test can reject $H_0$ or fail to reject it. It never proves $H_0$ true: failing to find evidence against a claim is not the same as establishing it.

One- and two-tailed tests

The wording of the question decides which:

A two-tailed test splits the significance level between both tails; a one-tailed test puts it all in one, which makes it easier to reject $H_0$ in that direction.

The significance level and critical region

The significance level $\alpha$ is how much evidence is demanded — conventionally $5\%$ or $1\%$. It is the probability of rejecting $H_0$ when it is actually true.

The critical region is the set of test statistic values that lead to rejection:

The z-test for a mean

When $\sigma$ is known (or $n$ is large):

$$z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}}$$

A machine is set to fill bottles to 500 ml with $\sigma = 20$. A sample of 40 bottles has mean 508 ml. Test at the 5% level whether the mean has changed.

1. Hypotheses. $H_0: \mu = 500$; $H_1: \mu \ne 500$ (two-tailed — "changed" gives no direction).

2. Significance level. $\alpha = 0.05$, critical values $\pm 1.96$.

3. Test statistic.

$$\text{SE} = \frac{20}{\sqrt{40}} = 3.162 \qquad z = \frac{508 - 500}{3.162} = 2.53$$

4. Compare. $2.53 > 1.96$, so the statistic falls in the critical region.

5. Conclude in context. Reject $H_0$: there is evidence at the 5% level that the mean fill volume has changed from 500 ml.

p-values

The p-value is the probability of a result at least as extreme as the one observed, assuming $H_0$ is true.

$$\text{Reject } H_0 \text{ if } p < \alpha$$

For the example above:

$$p = 2\big(1 - \Phi(2.53)\big) = 2(0.0057) = 0.0114$$

Since $0.0114 < 0.05$, reject $H_0$ — the same conclusion as the critical value method, as it must be.

The doubling is what makes it two-tailed: a result 2.53 standard errors below the claim would be equally surprising.

Type I and Type II errors

Type I — rejecting a true null hypothesis. A "false alarm". Its probability is exactly $\alpha$, which is why you choose $\alpha$: it is the false-alarm rate you are willing to accept.

Type II — failing to reject a false null hypothesis. A "missed detection". Its probability $\beta$ depends on how wrong $H_0$ actually is — a large discrepancy is easy to detect, a small one is not.

The power of a test is $1 - \beta$: the probability of correctly detecting a real effect.