Correlation and Linear Regression

Correlation measures how strongly two variables move together. Regression goes further and fits a line, so one variable can be predicted from the other. Both are easy to compute and easy to over-interpret.

Scatter diagrams

Plot the pairs $(x, y)$ before calculating anything. A scatter diagram shows at once whether the relationship is:

• positive — $y$ rises as $x$ rises;
• negative — $y$ falls as $x$ rises;
• linear or curved — and if it is curved, a correlation coefficient will understate it badly;
• affected by outliers — one distant point can dominate the whole calculation.

By convention $x$ is the explanatory (independent) variable and $y$ the response (dependent) variable.

The product-moment correlation coefficient

$$r = \frac{S_{xy}}{\sqrt{S_{xx}S_{yy}}}$$

where

$$S_{xy} = \sum xy - \frac{\left(\sum x\right)\left(\sum y\right)}{n} \qquad S_{xx} = \sum x^2 - \frac{\left(\sum x\right)^2}{n} \qquad S_{yy} = \sum y^2 - \frac{\left(\sum y\right)^2}{n}$$

For $x = 1,2,3,4,5$ and $y = 2,4,5,4,6$:

$$\sum x = 15,\ \sum y = 21,\ \sum xy = 71,\ \sum x^2 = 55,\ \sum y^2 = 97$$

$$S_{xy} = 71 - \frac{15 \times 21}{5} = 8 \qquad S_{xx} = 55 - 45 = 10 \qquad S_{yy} = 97 - 88.2 = 8.8$$

$$r = \frac{8}{\sqrt{10 \times 8.8}} = \frac{8}{9.381} = 0.853$$

Interpreting r

$$-1 \le r \le 1$$

So $r = 0.853$ indicates a fairly strong positive linear relationship.

Spearman's rank correlation

When data is ranked rather than measured — positions in a competition, quality ratings — or when the relationship is monotonic but not straight, use

$$r_s = 1 - \frac{6\sum d^2}{n(n^2-1)}$$

where $d$ is the difference between the two ranks for each item.

Two judges rank five entries. The rank differences are $-1, 1, -1, 1, 0$.

$$\sum d^2 = 1+1+1+1+0 = 4$$ $$r_s = 1 - \frac{6(4)}{5(24)} = 1 - \frac{24}{120} = 0.8$$

A strong measure of agreement between the judges.

The regression line

The least-squares regression line of $y$ on $x$ minimises the sum of the squared vertical distances from the points to the line:

$$y = a + bx \qquad b = \frac{S_{xy}}{S_{xx}} \qquad a = \bar{y} - b\bar{x}$$

For the data above:

$$b = \frac{8}{10} = 0.8 \qquad a = 4.2 - 0.8(3) = 1.8$$ $$y = 1.8 + 0.8x$$

The gradient $b$ is the practical result: each unit increase in $x$ is associated with an increase of $0.8$ in $y$.

The line always passes through $(\bar{x}, \bar{y})$. Here $1.8 + 0.8(3) = 4.2 = \bar{y}$ ✓ — a free check on the arithmetic.

Using the regression line

Interpolation — predicting inside the range of the data — is reasonable. For $x = 2.5$:

$$y = 1.8 + 0.8(2.5) = 3.8$$

Extrapolation — predicting outside it — is unreliable, however tidy the arithmetic. At $x = 20$ the line gives $y = 17.8$, but nothing in data spanning $x = 1$ to $5$ justifies assuming the pattern continues that far.