Every statistical conclusion is only as good as the data behind it. This topic is about getting data that actually represents what you want to know — and recognising the ways that goes wrong.
Populations and samples
A population is the entire set of items under study — every student in a country, every bolt from a factory.
A sample is a subset of it, chosen to stand in for the whole.
A census collects data from every member of the population.
| Advantages | Disadvantages | |
|---|---|---|
| Census | complete, no sampling error | expensive, slow, sometimes impossible |
| Sample | cheap, fast, feasible | subject to sampling error, may be unrepresentative |
A sampling frame is the list from which the sample is drawn. If the frame is incomplete — an old electoral roll, a phone directory that omits mobile-only households — the sample is compromised before it starts.
Sampling methods
Simple random sampling. Every member has an equal chance, and so does every possible sample. Number the population and use random numbers.
- Fair and unbiased; needs a complete sampling frame, and can by chance miss a small subgroup entirely.
Systematic sampling. Take every th member after a random start, where
For and : , so pick a random start between 1 and 20 and then take every 20th.
- Simple to carry out; but if the list has a hidden periodic pattern matching , the sample is badly skewed.
Stratified sampling. Divide the population into strata (groups sharing a characteristic) and sample each in proportion to its size:
A school has 500, 300 and 200 students in three years. Take a stratified sample of 100.
Totalling 100 ✓
- Guarantees every stratum is represented in proportion; needs the strata to be known in advance.
Cluster sampling. Divide into naturally occurring clusters (schools, districts, streets), pick some clusters at random, and survey everyone in them.
- Cheap when the population is geographically spread; less precise, since members of a cluster tend to resemble each other.
Quota sampling. The interviewer is told to find, say, 20 men and 20 women, choosing whoever they like within those quotas.
- Fast and needs no sampling frame; but it is not random, so the interviewer's choices can introduce serious bias, and no valid margin of error can be calculated.
Bias
Bias is any systematic tendency for a sample to misrepresent the population. Unlike random error, taking a bigger sample does not fix it.
Selection bias — the method favours some members. A street survey at 10am on a Tuesday over-represents people not at work.
Non-response bias — those who do not reply differ systematically from those who do. People with strong opinions are likelier to return a survey.
Response bias — the answers themselves are distorted, by a leading question ("Don't you agree that…"), by sensitive subject matter, or by respondents wanting to appear in a good light.
Sampling frame bias — the list omits part of the population.
Types of data
Qualitative data describes a category: colour, brand, opinion.
Quantitative data is numerical, and splits into:
- Discrete — counted, taking separate values. Number of children, goals scored.
- Continuous — measured, taking any value in a range. Height, time, mass.
Data can also be primary (collected by you, for your purpose) or secondary (already collected by someone else — cheaper and quicker, but you did not control how it was gathered or define its terms).
Collecting data
Observation — watching without intervening. Avoids the distortions of asking, but cannot reveal reasons.
Survey or questionnaire — asking directly. Cheap at scale, but exposed to response and non-response bias.
Experiment — actively changing a variable and measuring the effect. The only method that can establish cause.
A well-designed experiment has:
- a control group receiving no treatment, for comparison;
- randomisation of subjects to groups, so other differences average out;
- replication — enough subjects that chance variation does not dominate;
- where possible, blinding, so expectations do not colour the outcome.
Designing a questionnaire
Good questions are:
- clear and unambiguous — "How often do you exercise?" needs a defined period;
- not leading — "Don't you agree the service was excellent?" pushes the answer;
- exhaustive and mutually exclusive in their options — response categories –, – leave "10" in two places;
- relevant, and short enough that people finish;
- sensitive questions late, or made anonymous, so they do not cause abandonment.
A pilot survey on a small group is run first, to expose ambiguous wording and questions nobody can answer before the full survey is printed.