Contents / Statistics / Correlation and Regression

Chapter 7

Correlation and Regression

Covariance and correlation with the Cauchy-Schwarz bound; least squares derived from the normal equations; the variance decomposition and R-squared; the LINE conditions and the Gauss-Markov theorem; inference for the slope and the difference between confidence and prediction intervals; multiple regression in matrix form with the hat matrix as an orthogonal projection; multicollinearity and variance inflation; model selection and the nested F-test; and why a coefficient is not a causal effect.

Introduction

Every method so far has compared groups. Regression asks a different question: given a quantitative predictor, how does the response change as the predictor moves? The groups become a continuum, the $k$ means become a line, and almost everything from the previous chapter carries over — because, as that chapter's last section showed, ANOVA was already a regression.

The chapter has three movements. The first builds the simple case in scalar algebra, where every formula can be derived by hand: what correlation measures and what it provably cannot exceed, the least-squares line obtained by minimising a sum of squares, and the decomposition of variation that gives $R^{2}$ its meaning. The second adds inference — the sampling distribution of the slope, why the least-squares estimator is optimal among a large class, and the difference between predicting a mean and predicting an observation, which is the distinction most often collapsed in practice.

The third rewrites everything in matrix form and adds predictors. This is not a new theory but the same one written compactly, and the compression pays immediately: the normal equations become a single matrix equation, the fitted values become an orthogonal projection, and the residual-orthogonality facts that looked like lucky algebra in the scalar case become geometry. With that in hand, the genuinely multivariate problems — predictors that carry the same information, and choosing among models — can be stated properly.

A warning that the chapter returns to at the end: none of this establishes causation, and the mathematics is completely silent on the question. A regression coefficient is a statement about association in the data at hand, and converting it into a claim about intervention requires assumptions that no amount of fitting can supply.

Notation. $n$ paired observations $(x_i,y_i)$ , means $\bar x$ and $\bar y$ , and the shorthand sums

S_{xx} = \sum_i (x_i-\bar x)^2, \qquad S_{yy} = \sum_i (y_i-\bar y)^2, \qquad S_{xy} = \sum_i (x_i-\bar x)(y_i-\bar y).

7.1Correlation

Before fitting anything, ask how strongly two variables move together. The raw measure of co-movement is the covariance, and its defect is that it carries units.

Definition 7.1 (Covariance and correlation). The sample covariance and sample correlation of paired data are

s_{xy} = \frac{S_{xy}}{n-1}, \qquad r = \frac{S_{xy}}{\sqrt{S_{xx}S_{yy}}} = \frac{s_{xy}}{s_x s_y}.

The covariance of heights in centimetres against weights in kilograms changes if you switch to inches and pounds; the correlation does not. Dividing by both standard deviations strips the units, and the result is confined to a fixed range — which is the content of the next theorem and the reason $r$ is interpretable at all.

Theorem 7.2 (Correlation lies between -1 and 1). For any paired data with $S_{xx}, S_{yy} > 0$ ,

-1 \le r \le 1,

with $r = \pm 1$ if and only if the points lie exactly on a straight line of non-zero slope.

Proof. Write $u_i = x_i - \bar x$ and $v_i = y_i - \bar y$ . The Cauchy-Schwarz inequality for sums states

\left(\sum_i u_iv_i\right)^2 \le \left(\sum_i u_i^2\right)\left(\sum_i v_i^2\right),

that is $S_{xy}^2 \le S_{xx}S_{yy}$ . Dividing by the positive $S_{xx}S_{yy}$ gives $r^{2} \le 1$ , hence $|r|\le 1$ .

For the equality case, recall that Cauchy-Schwarz is an equality exactly when the two vectors are proportional: $v_i = cu_i$ for all $i$ and some constant $c \ne 0$ . That says $y_i - \bar y = c(x_i - \bar x)$ for every $i$ , which is the equation of a line through $(\bar x,\bar y)$ with slope $c$ . The sign of $r$ matches the sign of $c$ .∎

Proposition 7.3 (Correlation is invariant under increasing linear rescaling). If $x_i' = a + bx_i$ and $y_i' = c + dy_i$ with $b,d \ne 0$ , then $r' = \operatorname{sign}(bd)\,r$ . In particular $r$ is unchanged when both scales are multiplied by positive constants or shifted.

Proof. Centring removes the additive constants: $x_i'-\bar x' = b(x_i-\bar x)$ and similarly for $y$ . Hence $S_{xy}' = bd\,S_{xy}$ , $S_{xx}' = b^{2}S_{xx}$ and $S_{yy}' = d^{2}S_{yy}$ . Substituting,

r' = \frac{bd\,S_{xy}}{\sqrt{b^2S_{xx}\,d^2S_{yy}}} = \frac{bd}{|b||d|}\cdot r = \operatorname{sign}(bd)\,r.

∎

Pitfall. $r$ measures linear association and nothing else. Data lying exactly on the parabola $y=x^{2}$ over a symmetric range of $x$ has $r=0$ — a perfect, deterministic relationship that the correlation reports as no association at all.

The converse failure is equally common: a single outlying point can move $r$ from near zero to near one, since every term in $S_{xy}$ is a product of two deviations and an extreme point contributes an enormous one. Always plot the data. A correlation quoted without a scatterplot is a number whose meaning has not been checked.

Example 7.4 (Computing r by hand). For the data $x = 1,2,3,4,5$ and $y = 2,4,5,4,5$ , compute $r$ .

Solution.

Means: $\bar x = 3$ , $\bar y = 4$ .
Deviations: $u = -2,-1,0,1,2$ and $v = -2,0,1,0,1$ .
$S_{xx} = 4+1+0+1+4 = 10$ ; $S_{yy} = 4+0+1+0+1 = 6$ ; $S_{xy} = 4+0+0+0+2 = 6$ .
Therefore

r = \frac{6}{\sqrt{10\times 6}} = \frac{6}{\sqrt{60}} = \frac{6}{7.74597} = 0.7746 .

Sanity check: $|r| \le 1$ ✓ as the theorem requires, and the value is positive, matching the generally upward drift of the data. Note it is not close to $1$ despite the upward trend — the point $(4,4)$ breaks the pattern, and $r$ registers that.

□

Example 7.5 (One point moving the correlation). To the data $x=1,2,3,4,5$ , $y=2,4,5,4,5$ (with $r=0.7746$ ) add a sixth point $(20,25)$ . Recompute $r$ .

Solution.

New means: $\bar x = 35/6 = 5.8333$ , $\bar y = 45/6 = 7.5$ .
The new point's deviations are $u_6 = 14.1667$ and $v_6 = 17.5$ , contributing $14.1667\times 17.5 = 247.92$ to $S_{xy}$ — against a total of $6$ from all five original points.
Computing the full sums: $S_{xx} = 250.83$ , $S_{yy} = 373.50$ , $S_{xy} = 303.50$ . Hence

r = \frac{303.50}{\sqrt{250.83\times 373.50}} = \frac{303.50}{306.08} = 0.9916 .

A single point has moved the correlation from $0.77$ to $0.99$ , and it contributes $247.92/303.50 = 82\%$ of $S_{xy}$ by itself.
Sanity check on why: every term of $S_{xy}$ is a product of two deviations, so a point far from both means contributes quadratically. One observation at ten times the spread of the others dominates the sum entirely.
The reported $r=0.99$ is not describing the five original points at all — it is describing the fact that one point sits far away in a consistent direction. This is why the pitfall above insists on a scatterplot.

□

Exercises for §7.1 →

7.2The least-squares line

Correlation summarises association with one number. A regression goes further and produces a function, which can be used to predict.

Definition 7.6 (The simple linear regression model). The model treats the predictor as fixed and the response as random,

Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \qquad \varepsilon_i \stackrel{\text{iid}}{\sim} N(0,\sigma^2),

so that $E(Y_i) = \beta_0+\beta_1x_i$ and each observation deviates from that line by independent noise of constant variance.

Theorem 7.7 (The least-squares estimates). The values of $b_0,b_1$ minimising the residual sum of squares $Q(b_0,b_1) = \sum_i (y_i - b_0 - b_1x_i)^2$ are

b_1 = \frac{S_{xy}}{S_{xx}}, \qquad b_0 = \bar y - b_1\bar x,

and this minimum is unique whenever $S_{xx}>0$ .

Proof. $Q$ is a quadratic in two variables with positive-definite Hessian, so setting both partial derivatives to zero locates the unique global minimum.

Differentiating in $b_0$ :

\frac{\partial Q}{\partial b_0} = -2\sum_i (y_i - b_0 - b_1x_i) = 0 \iff \sum_i y_i = nb_0 + b_1\sum_i x_i,

which on dividing by $n$ gives $\bar y = b_0 + b_1\bar x$ — the line passes through the point of means.

Differentiating in $b_1$ :

\frac{\partial Q}{\partial b_1} = -2\sum_i x_i(y_i-b_0-b_1x_i) = 0 \iff \sum_i x_iy_i = b_0\sum_i x_i + b_1\sum_i x_i^2 .

These two are the normal equations. Substitute $b_0 = \bar y - b_1\bar x$ into the second and rearrange:

\sum_i x_iy_i - n\bar x\bar y = b_1\left(\sum_i x_i^2 - n\bar x^2\right),

and the two sides are precisely $S_{xy}$ and $b_1S_{xx}$ , by the computational identity of the theorem Computational formula for the variance. Hence $b_1 = S_{xy}/S_{xx}$ .

For uniqueness, the Hessian of $Q$ is $2\begin{pmatrix} n & \sum x_i \\ \sum x_i & \sum x_i^2\end{pmatrix}$ , whose determinant is $4n\big(\sum x_i^{2} - n\bar x^{2}\big) = 4nS_{xx} > 0$ ; with positive diagonal it is positive definite, so the stationary point is a strict minimum.∎

Corollary 7.8 (Slope, correlation and the regression effect). The slope and the correlation are related by

b_1 = r\,\frac{s_y}{s_x}.

Proof. $b_1 = S_{xy}/S_{xx}$ , and $r = S_{xy}/\sqrt{S_{xx}S_{yy}}$ , so $S_{xy} = r\sqrt{S_{xx}S_{yy}}$ . Substituting,

b_1 = \frac{r\sqrt{S_{xx}S_{yy}}}{S_{xx}} = r\sqrt{\frac{S_{yy}}{S_{xx}}} = r\frac{s_y}{s_x}.

∎

Intuition. That corollary contains the phenomenon that gave regression its name. Measure both variables in standard units and the slope is $r$ , which has absolute value at most $1$ . So a subject one standard deviation above average in $x$ is predicted to be only $r$ standard deviations above average in $y$ — closer to the mean, in relative terms, than they were.

This is regression to the mean, and it is a property of imperfect correlation, not a force acting on anything. Tall fathers have sons who are tall but on average less exceptionally so; a season of exceptional performance is typically followed by a more ordinary one. Nothing is pulling anyone back: the second measurement simply contains its own independent noise, and an extreme first measurement was probably extreme partly by luck.

Proposition 7.9 (Two properties of least-squares residuals). With $e_i = y_i - b_0 - b_1x_i$ the residuals from the least-squares fit,

\sum_i e_i = 0 \qquad \text{and} \qquad \sum_i x_ie_i = 0 .

Proof. These are exactly the two normal equations of the previous proof, rewritten. The first derivative condition says $\sum_i(y_i-b_0-b_1x_i)=0$ , which is $\sum_i e_i = 0$ ; the second says $\sum_i x_i(y_i-b_0-b_1x_i)=0$ , which is $\sum_i x_ie_i=0$ .∎

The second identity says the residuals are orthogonal to the predictor: whatever structure is left over carries no remaining linear relationship with $x$ . That is a consequence of how the line was chosen, not a property of the data, and it is why a plot of residuals against $x$ can never show a linear trend — only curvature or changing spread, which is exactly what makes such a plot a useful diagnostic.

Example 7.10 (Fitting the line). Fit the least-squares line to $x = 1,2,3,4,5$ , $y = 2,4,5,4,5$ , and predict $y$ at $x=6$ .

Solution.

From the correlation example, $S_{xx}=10$ , $S_{xy}=6$ , $\bar x = 3$ , $\bar y = 4$ .
Slope: $b_1 = 6/10 = 0.6$ . Intercept: $b_0 = 4 - 0.6(3) = 4-1.8 = 2.2$ .
The fitted line is $\hat y = 2.2 + 0.6x$ .
Check against the corollary Slope, correlation and the regression effect: $s_y/s_x = \sqrt{6/10} = 0.7746$ , and $r\,s_y/s_x = 0.7746 \times 0.7746 = 0.6$ ✓.
Fitted values at $x=1,\ldots,5$ : $2.8,\ 3.4,\ 4.0,\ 4.6,\ 5.2$ . Residuals: $-0.8,\ 0.6,\ 1.0,\ -0.6,\ -0.2$ , which sum to $0$ ✓ as the proposition requires.
Prediction at $x=6$ : $\hat y = 2.2+3.6 = 5.8$ . But $x=6$ is outside the observed range $[1,5]$ , so this is extrapolation — see the last section for why that number deserves less confidence than the arithmetic suggests.

□

Exercises for §7.2 →

7.3Residuals, R-squared, and what the fit explains

The same decomposition that drove ANOVA reappears here, and for the same reason: the fitted values play the role the group means played.

Theorem 7.11 (Decomposition of variation in regression). With $SST = S_{yy}$ , $SSR = \sum_i(\hat y_i - \bar y)^2$ and $SSE = \sum_i (y_i-\hat y_i)^2$ ,

SST = SSR + SSE .

Proof. Write $y_i - \bar y = (\hat y_i - \bar y) + (y_i - \hat y_i)$ , square and sum:

SST = SSR + SSE + 2\sum_i (\hat y_i-\bar y)(y_i-\hat y_i) .

The cross term vanishes. Since $\hat y_i = b_0+b_1x_i$ and $\bar y = b_0 + b_1\bar x$ , we have $\hat y_i - \bar y = b_1(x_i-\bar x)$ , and $y_i - \hat y_i = e_i$ . So the cross term is

2b_1\sum_i (x_i-\bar x)e_i = 2b_1\left(\sum_i x_ie_i - \bar x\sum_i e_i\right) = 2b_1(0 - 0) = 0

by the proposition Two properties of least-squares residuals.∎

Definition 7.12 (Coefficient of determination). The proportion of variation in $y$ accounted for by the fit is

R^2 = \frac{SSR}{SST} = 1 - \frac{SSE}{SST} \in [0,1].

Proposition 7.13 (For simple regression, R-squared is r squared). In simple linear regression, $R^{2} = r^{2}$ , and $SSR = b_1 S_{xy} = b_1^2 S_{xx}$ .

Proof. Using $\hat y_i - \bar y = b_1(x_i-\bar x)$ ,

SSR = b_1^2\sum_i (x_i-\bar x)^2 = b_1^2 S_{xx} = \frac{S_{xy}^2}{S_{xx}^2}\cdot S_{xx} = \frac{S_{xy}^2}{S_{xx}},

which also equals $b_1 S_{xy}$ . Dividing by $SST = S_{yy}$ ,

R^2 = \frac{S_{xy}^2}{S_{xx}S_{yy}} = r^2 .

∎

Definition 7.14 (Residual standard error). The error variance is estimated by

s^2 = \frac{SSE}{n-2},

the divisor being $n-2$ because two parameters were estimated from the data.

Pitfall. $R^{2}$ measures how much variation the fit accounts for, and that is all. It does not say the model is correct, that the relationship is linear, or that $x$ causes $y$ .

Anscombe's quartet is the standard demonstration: four data sets with identical means, identical variances, identical $r$ , identical fitted lines and identical $R^{2}$ , of which only one is appropriately described by a straight line — the others contain a clear curve, a single high-leverage point driving everything, and an outlier. Every summary statistic agrees; the scatterplots do not.

$R^{2}$ also rises mechanically with every predictor added, which is what the adjusted version in the model-selection section corrects.

Example 7.15 (Decomposing the variation). For the fitted line $\hat y = 2.2+0.6x$ on the running example, compute $SSR$ , $SSE$ , $R^{2}$ and $s$ .

Solution.

By the proposition For simple regression, R-squared is r squared, $SSR = b_1S_{xy} = 0.6\times 6 = 3.6$ .
$SST = S_{yy} = 6$ , so $SSE = 6 - 3.6 = 2.4$ .
Confirm directly from the residuals $-0.8, 0.6, 1.0, -0.6, -0.2$ : their squares are $0.64, 0.36, 1.00, 0.36, 0.04$ , summing to $2.40$ ✓.
$R^{2} = 3.6/6 = 0.6$ , and indeed $r^{2} = 0.7746^2 = 0.6$ ✓.
$s^{2} = SSE/(n-2) = 2.4/3 = 0.8$ , so $s = 0.894$ .
Reading the result: the line accounts for $60\%$ of the variation in $y$ , and a typical observation sits about $0.89$ units off it. With $\bar y = 4$ , that residual spread is not small relative to the response, which is the honest counterweight to a respectable-sounding $R^{2}$ .

□

Figure 7.16. The fitted line

\hat y = 2.2+0.6x

with the five data points and their residuals drawn as vertical segments. Least squares chooses the line minimising the total squared length of those segments — not their total length, and not the perpendicular distances. The dashed horizontal line is

\bar y = 4

, the prediction the model would make with no predictor at all;

R^{2}

compares the squared lengths against that baseline.

Exercises for §7.3 →

7.4The conditions, and why least squares is optimal

Definition 7.17 (The LINE conditions). The simple regression model assumes

Linearity — $E(Y\mid x)$ really is $\beta_0+\beta_1x$ ;
Independence — the errors are independent of one another;
Normality — the errors are normally distributed;
Equal variance — $\operatorname{Var}(\varepsilon_i)=\sigma^{2}$ for every $i$ (homoscedasticity).

These are not equally important, and the following theorem says exactly which ones the central result needs. Notice what is absent from its hypotheses.

Theorem 7.18 (Gauss-Markov). Assume linearity, independence and equal variance, with $E(\varepsilon_i)=0$ — normality is not required. Then among all estimators of $\beta_1$ that are linear in the $Y_i$ and unbiased, the least-squares estimator $b_1$ has the smallest variance. It is the best linear unbiased estimator.

Proof. First write $b_1$ in linear form. Since $\sum_i(x_i-\bar x)\bar y = 0$ ,

b_1 = \frac{S_{xy}}{S_{xx}} = \sum_i w_iY_i \qquad \text{with} \qquad w_i = \frac{x_i-\bar x}{S_{xx}},

and these weights satisfy two identities used repeatedly below:

\sum_i w_i = 0, \qquad \sum_i w_ix_i = \frac{\sum_i (x_i-\bar x)x_i}{S_{xx}} = \frac{S_{xx}}{S_{xx}} = 1 .

Unbiasedness. $E(b_1) = \sum_i w_i(\beta_0+\beta_1x_i) = \beta_0\sum_i w_i + \beta_1\sum_i w_ix_i = 0 + \beta_1 = \beta_1$ .

Variance. By independence and equal variance, $\operatorname{Var}(b_1) = \sigma^{2}\sum_i w_i^{2} = \sigma^{2} S_{xx}/S_{xx}^2 = \sigma^{2}/S_{xx}$ .

Optimality. Let $\tilde b = \sum_i c_iY_i$ be any other linear unbiased estimator. Unbiasedness for every $\beta_0,\beta_1$ forces $\sum_i c_i = 0$ and $\sum_i c_ix_i = 1$ , by the same expansion. Write $c_i = w_i + d_i$ ; then the two constraints give $\sum_i d_i = 0$ and $\sum_i d_ix_i = 0$ . Now

\sum_i c_i^2 = \sum_i w_i^2 + 2\sum_i w_id_i + \sum_i d_i^2,

and the cross term vanishes:

\sum_i w_id_i = \frac{1}{S_{xx}}\sum_i (x_i-\bar x)d_i = \frac{1}{S_{xx}}\left(\sum_i x_id_i - \bar x\sum_i d_i\right) = 0 .

Therefore $\operatorname{Var}(\tilde b) = \sigma^{2}\sum_i c_i^{2} = \sigma^{2}\big(\sum_i w_i^{2} + \sum_i d_i^{2}\big) \ge \sigma^{2}\sum_i w_i^{2} = \operatorname{Var}(b_1)$ , with equality only when every $d_i=0$ , that is $\tilde b = b_1$ .∎

The theorem is doing more than it appears. It says least squares is not an arbitrary choice of loss function: under three assumptions that say nothing about the shape of the error distribution, minimising squared error delivers the best estimator in a large and natural class. Normality is needed only later, for the $t$ -distributions used in inference — and even there the central limit theorem makes the requirement soft for large $n$ .

Corollary 7.19 (Precision of the slope). $\operatorname{Var}(b_1) = \sigma^{2}/S_{xx}$ , estimated by $s^{2}/S_{xx}$ , so the standard error of the slope is $s/\sqrt{S_{xx}}$ .

Proof. Contained in the proof above; the estimate substitutes $s^{2}$ for $\sigma^{2}$ .∎

Remark (How to make a slope precise). $S_{xx}$ in the denominator is the design lever. It grows with the sample size and with the spread of the predictor values, so a slope is estimated precisely when the $x$ 's are spread widely, and imprecisely when they are bunched together however many there are.

That has a concrete design consequence: if the linear model is trusted and the only goal is estimating the slope, the optimal allocation puts half the observations at each extreme of the feasible $x$ range. The reason not to do that in practice is that such a design has no ability whatever to detect curvature — with two distinct $x$ values, every fit is a perfect straight line through two means. Spreading $x$ buys precision; keeping intermediate values buys the ability to check the assumption.

Pitfall. Of the four conditions, independence is again the one that cannot be repaired after the fact and cannot be checked from the fitted values alone. Time-ordered data with correlated errors produces standard errors that are too small — sometimes by a large factor — so slopes look significant when they are not.

Equal variance is checked by plotting residuals against fitted values and looking for a funnel shape. A widening funnel does not bias $b_1$ , which stays unbiased by Gauss-Markov's independence of the normality assumption, but it invalidates the standard errors and therefore every $p$ -value and interval.

Linearity is checked on the same plot, looking for curvature. A residual plot showing a clear U shape means the model is wrong in a way that $R^{2}$ may not reveal.

Figure 7.20. Three residual-versus-fitted plots. The first shows what a satisfied model looks like: residuals scattered in a horizontal band of constant width, centred on zero, with no pattern. The second funnels outward — the variance grows with the fitted value, so equal variance fails and every standard error is wrong. The third bends — the true relationship is curved, so linearity fails and the fitted line is systematically too low at the ends and too high in the middle. Only the first licenses the inference in this chapter.

Example 7.21 (Reading a residual plot). A regression of sales on advertising spend gives $R^{2} = 0.88$ , and the residual plot funnels outward. What is compromised, and what is not?

Solution.

Still valid. By the theorem Gauss-Markov... almost. Gauss-Markov assumes equal variance, so strictly the optimality claim goes too. But unbiasedness of $b_1$ survives — it needs only $E(\varepsilon_i)=0$ and linearity, neither of which a funnel violates. The fitted line is still estimating the right slope on average.
Compromised. Every standard error, every $t$ statistic, every confidence and prediction interval. All of them were computed from a single $\sigma^{2}$ , and there is no single $\sigma^{2}$ .
The direction of the error is not uniform: the intervals are too narrow where the spread is large and too wide where it is small, so the overall coverage can look roughly right while being wrong at every individual $x$ .
Remedies: model the variance (weighted least squares, weighting each observation by $1/\hat\sigma_i^{2}$ ), transform the response (a log transform often stabilises a funnel that widens proportionally), or use standard errors robust to heteroscedasticity.
Sanity check on the role of $R^{2} = 0.88$ : it says nothing about this. $R^{2}$ is computed from the sums of squares and is entirely blind to how the residuals are distributed across the range — which is exactly the pitfall about Anscombe's quartet, met here in the wild.

□

Exercises for §7.4 →

7.5Inference for the slope, and two kinds of prediction

Theorem 7.22 (Sampling distribution of the slope). Under the full LINE conditions,

\frac{b_1 - \beta_1}{s/\sqrt{S_{xx}}} \sim t_{\,n-2},

so a $100(1-\alpha)\%$ confidence interval for $\beta_1$ is $b_1 \pm t_{\alpha/2,n-2}\,s/\sqrt{S_{xx}}$ , and $H_0:\beta_1=0$ is tested by referring $b_1/(s/\sqrt{S_{xx}})$ to $t_{n-2}$ .

Proof. By the theorem Gauss-Markov, $b_1$ is a linear combination of the $Y_i$ ; under normality that combination is normal, with mean $\beta_1$ and variance $\sigma^{2}/S_{xx}$ . So $(b_1-\beta_1)/(\sigma/\sqrt{S_{xx}})$ is standard normal.

Independently of it, $SSE/\sigma^{2} \sim \chi^{2}_{n-2}$ — quoted, as the analogous fact was in the $t$ -test chapter, the degrees of freedom being $n-2$ because two parameters were fitted. Forming the ratio $Z/\sqrt{V/(n-2)}$ as in the definition Student's t-distribution replaces $\sigma$ by $s$ and gives $t_{n-2}$ .∎

Testing $H_0:\beta_1=0$ is testing whether the predictor carries any linear information about the response. For simple regression this is the same test as the ANOVA $F$ of the previous chapter, and indeed $F = t^{2}$ again — the model comparison being between the fitted line and the intercept-only model.

Prediction needs more care, because there are two different questions and they have different answers.

Theorem 7.23 (Confidence and prediction intervals). At a predictor value $x_0$ , with $\hat y_0 = b_0+b_1x_0$ , a $100(1-\alpha)\%$ interval for the mean response $E(Y\mid x_0)$ is

\hat y_0 \pm t_{\alpha/2,n-2}\;s\sqrt{\frac{1}{n} + \frac{(x_0-\bar x)^2}{S_{xx}}},

while a $100(1-\alpha)\%$ prediction interval for a single new observation at $x_0$ is

\hat y_0 \pm t_{\alpha/2,n-2}\;s\sqrt{1 + \frac{1}{n} + \frac{(x_0-\bar x)^2}{S_{xx}}}.

Proof. Write $\hat y_0 = \bar y + b_1(x_0-\bar x)$ , using $b_0 = \bar y - b_1\bar x$ . The two terms are independent — $\bar Y$ and $b_1$ are uncorrelated, since $\operatorname{Cov}(\bar Y, b_1) = \sigma^{2}\sum_i w_i/n = 0$ by the weight identity — so

\operatorname{Var}(\hat y_0) = \frac{\sigma^2}{n} + (x_0-\bar x)^2\frac{\sigma^2}{S_{xx}} = \sigma^2\left(\frac1n + \frac{(x_0-\bar x)^2}{S_{xx}}\right),

which gives the first interval on replacing $\sigma$ by $s$ and using $t_{n-2}$ .

For a new observation $Y_0 = \beta_0+\beta_1x_0+\varepsilon_0$ , the prediction error is $Y_0 - \hat y_0$ , and $\varepsilon_0$ is independent of the data used to fit. Variances therefore add:

\operatorname{Var}(Y_0 - \hat y_0) = \sigma^2 + \operatorname{Var}(\hat y_0) = \sigma^2\left(1 + \frac1n + \frac{(x_0-\bar x)^2}{S_{xx}}\right),

which is the second interval.∎

Intuition. The extra $1$ under the square root is the whole difference, and it never goes away. Estimating the average response at $x_0$ gets easier with more data, and the interval shrinks toward zero width as $n\to\infty$ . Predicting a single new observation does not, because that observation has its own noise $\varepsilon_0$ , which no amount of data about other observations can reduce.

So the prediction interval has a floor at roughly $\pm t\,s$ , and with infinite data it converges to the interval you would draw around the true line from knowing $\sigma$ alone. Confusing the two produces prediction intervals far too narrow, which is one of the more consequential errors in applied work.

Example 7.24 (Both intervals, at the centre and at the edge). For the running example ( $n=5$ , $s=0.894$ , $S_{xx}=10$ , $\bar x = 3$ , $\hat y = 2.2+0.6x$ , $t_{0.025,3}=3.182$ ), give the $95\%$ confidence and prediction intervals at $x_0=3$ and at $x_0=5$ .

Solution.

At $x_0=3=\bar x$ . $\hat y_0 = 2.2+1.8 = 4.0$ , and $(x_0-\bar x)^2 = 0$ .
Mean response: $s\sqrt{1/5} = 0.894\times 0.4472 = 0.400$ , margin $3.182\times 0.400 = 1.273$ , interval $(2.73,\ 5.27)$ .
New observation: $s\sqrt{1+1/5} = 0.894\times 1.0954 = 0.980$ , margin $3.182\times 0.980 = 3.118$ , interval $(0.88,\ 7.12)$ — nearly two and a half times as wide.
At $x_0=5$ . $\hat y_0 = 5.2$ , and $(x_0-\bar x)^2/S_{xx} = 4/10 = 0.4$ .
Mean response: $s\sqrt{0.2+0.4} = 0.894\times 0.7746 = 0.693$ , margin $2.204$ , interval $(3.00,\ 7.40)$ .
Sanity check on the shape: both intervals are narrowest at $\bar x$ and widen as $x_0$ moves away, because the $(x_0-\bar x)^2$ term grows. The fitted line pivots about the point of means, so a small error in the slope has little effect near $\bar x$ and a growing effect further out — which is the mathematical reason extrapolation is dangerous, quantified.

□

Example 7.25 (Testing the slope). Test $H_0:\beta_1=0$ for the running example at $\alpha = 0.05$ .

Solution.

Standard error: $s/\sqrt{S_{xx}} = 0.894/\sqrt{10} = 0.894/3.1623 = 0.2828$ .
Statistic: $t = 0.6/0.2828 = 2.121$ on $n-2 = 3$ degrees of freedom.
Against $t_{0.025,3} = 3.182$ : do not reject. The slope is not distinguishable from zero.
Sanity check against $R^{2}$ : the fit accounts for $60\%$ of the variation, which sounds substantial, yet the slope is not significant — because $n=5$ leaves only $3$ degrees of freedom and the critical value is correspondingly large. A high $R^{2}$ on tiny data is not evidence of a real relationship.
Confirm via $F$ : $F = t^{2} = 4.5$ , and $MSR/MSE = 3.6/(2.4/3) = 3.6/0.8 = 4.5$ ✓, the equivalence noted after the theorem Sampling distribution of the slope.
The confidence interval says the same thing more usefully: $0.6 \pm 3.182(0.2828) = 0.6\pm 0.900 = (-0.30,\ 1.50)$ . It contains zero, and it is wide enough to be compatible with a substantial positive slope as well — the experiment has not settled the question either way.

□

Figure 7.26. Confidence and prediction bands for the fitted line. Both are narrowest at

\bar x = 3

and flare outwards, because the fitted line pivots about the point of means. The prediction band (outer) stays far wider than the confidence band (inner) at every

x

, and the gap never closes: more data shrinks the uncertainty about the line but not the noise in a future observation.

Exercises for §7.5 →

7.6Many predictors: the matrix formulation

With several predictors the scalar formulas become unmanageable, and rewriting in matrix notation does more than save ink: it turns the algebra into geometry, and the geometry explains results that looked accidental.

Definition 7.27 (The multiple regression model in matrix form). With $p$ parameters (an intercept and $p-1$ predictors), collect the data into

\mathbf{y} = \begin{pmatrix} y_1\\ \vdots\\ y_n\end{pmatrix}, \qquad \mathbf{X} = \begin{pmatrix} 1 & x_{11} & \cdots & x_{1,p-1}\\ \vdots & \vdots & & \vdots \\ 1 & x_{n1} & \cdots & x_{n,p-1}\end{pmatrix}, \qquad \boldsymbol\beta = \begin{pmatrix}\beta_0\\ \vdots\\ \beta_{p-1}\end{pmatrix},

with $\mathbf{X}$ the design matrix. The model is

\mathbf{y} = \mathbf{X}\boldsymbol\beta + \boldsymbol\varepsilon, \qquad E(\boldsymbol\varepsilon)=\mathbf{0},\quad \operatorname{Var}(\boldsymbol\varepsilon)=\sigma^2\mathbf{I}.

The single equation $\operatorname{Var}(\boldsymbol\varepsilon)=\sigma^{2}\mathbf{I}$ encodes two of the LINE conditions at once: the off-diagonal zeros are independence, and the constant diagonal is equal variance.

Theorem 7.28 (The normal equations and the least-squares solution). The vector minimising $\|\mathbf{y}-\mathbf{X}\mathbf{b}\|^2$ satisfies the normal equations

\mathbf{X}^{\!\top}\mathbf{X}\,\hat{\boldsymbol\beta} = \mathbf{X}^{\!\top}\mathbf{y},

and if $\mathbf{X}$ has full column rank then $\mathbf{X}^{\!\top}\mathbf{X}$ is invertible and

\hat{\boldsymbol\beta} = (\mathbf{X}^{\!\top}\mathbf{X})^{-1}\mathbf{X}^{\!\top}\mathbf{y}.

Proof. Expand the objective:

Q(\mathbf{b}) = (\mathbf{y}-\mathbf{X}\mathbf{b})^{\!\top}(\mathbf{y}-\mathbf{X}\mathbf{b}) = \mathbf{y}^{\!\top}\mathbf{y} - 2\mathbf{b}^{\!\top}\mathbf{X}^{\!\top}\mathbf{y} + \mathbf{b}^{\!\top}\mathbf{X}^{\!\top}\mathbf{X}\mathbf{b},

using $\mathbf{b}^{\!\top}\mathbf{X}^{\!\top}\mathbf{y} = \mathbf{y}^{\!\top}\mathbf{X}\mathbf{b}$ , both being scalars. Differentiating with respect to $\mathbf{b}$ ,

\nabla Q = -2\mathbf{X}^{\!\top}\mathbf{y} + 2\mathbf{X}^{\!\top}\mathbf{X}\mathbf{b},

and setting it to zero gives the normal equations.

For full rank: if $\mathbf{X}^{\!\top}\mathbf{X}\mathbf{v}=\mathbf{0}$ then $\mathbf{v}^{\!\top}\mathbf{X}^{\!\top}\mathbf{X}\mathbf{v} = \|\mathbf{X}\mathbf{v}\|^2 = 0$ , so $\mathbf{X}\mathbf{v}=\mathbf{0}$ , and full column rank forces $\mathbf{v}=\mathbf{0}$ . Hence $\mathbf{X}^{\!\top}\mathbf{X}$ is invertible, and being positive definite by the same computation, the stationary point is the unique minimum.∎

Definition 7.29 (Hat matrix). The hat matrix is $\mathbf{H} = \mathbf{X}(\mathbf{X}^{\!\top}\mathbf{X})^{-1}\mathbf{X}^{\!\top}$ , so that the fitted values and residuals are

\hat{\mathbf{y}} = \mathbf{X}\hat{\boldsymbol\beta} = \mathbf{H}\mathbf{y}, \qquad \mathbf{e} = \mathbf{y}-\hat{\mathbf{y}} = (\mathbf{I}-\mathbf{H})\mathbf{y}.

Theorem 7.30 (Least squares is orthogonal projection). $\mathbf{H}$ is symmetric and idempotent ( $\mathbf{H}^2=\mathbf{H}$ ), satisfies $\mathbf{H}\mathbf{X}=\mathbf{X}$ , and is the orthogonal projection onto the column space of $\mathbf{X}$ . Consequently $\mathbf{X}^{\!\top}\mathbf{e}=\mathbf{0}$ : the residual vector is orthogonal to every predictor.

Proof. Symmetry: $\mathbf{H}^{\!\top} = \mathbf{X}\big((\mathbf{X}^{\!\top}\mathbf{X})^{-1}\big)^{\!\top}\mathbf{X}^{\!\top} = \mathbf{H}$ , since $\mathbf{X}^{\!\top}\mathbf{X}$ is symmetric and so is its inverse.

Idempotence: the inner factors cancel,

\mathbf{H}^2 = \mathbf{X}(\mathbf{X}^{\!\top}\mathbf{X})^{-1}\underbrace{\mathbf{X}^{\!\top}\mathbf{X}(\mathbf{X}^{\!\top}\mathbf{X})^{-1}}_{=\;\mathbf{I}}\mathbf{X}^{\!\top} = \mathbf{H}.

Likewise $\mathbf{H}\mathbf{X} = \mathbf{X}(\mathbf{X}^{\!\top}\mathbf{X})^{-1}\mathbf{X}^{\!\top}\mathbf{X} = \mathbf{X}$ , so $\mathbf{H}$ fixes the column space pointwise. A symmetric idempotent matrix fixing a subspace pointwise is the orthogonal projection onto it.

Finally, from the normal equations $\mathbf{X}^{\!\top}\mathbf{X}\hat{\boldsymbol\beta}=\mathbf{X}^{\!\top}\mathbf{y}$ we get

\mathbf{X}^{\!\top}\mathbf{e} = \mathbf{X}^{\!\top}(\mathbf{y}-\mathbf{X}\hat{\boldsymbol\beta}) = \mathbf{X}^{\!\top}\mathbf{y} - \mathbf{X}^{\!\top}\mathbf{X}\hat{\boldsymbol\beta} = \mathbf{0}.

∎

Intuition. The geometry makes the whole method obvious in hindsight. The vector $\mathbf{y}$ lives in $\mathbb{R}^n$ ; the fitted values $\mathbf{X}\mathbf{b}$ , as $\mathbf{b}$ ranges over all coefficient vectors, sweep out a $p$ -dimensional subspace — the column space of $\mathbf{X}$ , the set of all predictions the model is capable of making.

Least squares asks for the point of that subspace closest to $\mathbf{y}$ , and the closest point of a subspace is the foot of the perpendicular. So the fit is the orthogonal projection, and the residual is the perpendicular component. Every algebraic identity in this chapter is that picture written in coordinates: residuals summing to zero is orthogonality to the intercept column of ones, $\sum x_ie_i=0$ is orthogonality to the predictor column, and $SST=SSR+SSE$ is Pythagoras' theorem on the right triangle with legs $\hat{\mathbf{y}}-\bar y\mathbf{1}$ and $\mathbf{e}$ .

Theorem 7.31 (Mean and variance of the coefficient estimates). Under the model, $\hat{\boldsymbol\beta}$ is unbiased with

E(\hat{\boldsymbol\beta}) = \boldsymbol\beta, \qquad \operatorname{Var}(\hat{\boldsymbol\beta}) = \sigma^2(\mathbf{X}^{\!\top}\mathbf{X})^{-1},

and $s^{2} = SSE/(n-p)$ is unbiased for $\sigma^{2}$ . The standard error of $\hat\beta_j$ is $s\sqrt{[(\mathbf{X}^{\!\top}\mathbf{X})^{-1}]_{jj}}$ , and $(\hat\beta_j-\beta_j)/SE(\hat\beta_j) \sim t_{\,n-p}$ .

Proof. Write $\mathbf{A} = (\mathbf{X}^{\!\top}\mathbf{X})^{-1}\mathbf{X}^{\!\top}$ , a fixed matrix, so $\hat{\boldsymbol\beta} = \mathbf{A}\mathbf{y}$ is linear in $\mathbf{y}$ . Then

E(\hat{\boldsymbol\beta}) = \mathbf{A}E(\mathbf{y}) = \mathbf{A}\mathbf{X}\boldsymbol\beta = (\mathbf{X}^{\!\top}\mathbf{X})^{-1}\mathbf{X}^{\!\top}\mathbf{X}\boldsymbol\beta = \boldsymbol\beta .

For the variance, the rule $\operatorname{Var}(\mathbf{A}\mathbf{y}) = \mathbf{A}\operatorname{Var}(\mathbf{y})\mathbf{A}^{\!\top}$ with $\operatorname{Var}(\mathbf{y})=\sigma^{2}\mathbf{I}$ gives

\operatorname{Var}(\hat{\boldsymbol\beta}) = \sigma^2\mathbf{A}\mathbf{A}^{\!\top} = \sigma^2(\mathbf{X}^{\!\top}\mathbf{X})^{-1}\mathbf{X}^{\!\top}\mathbf{X}(\mathbf{X}^{\!\top}\mathbf{X})^{-1} = \sigma^2(\mathbf{X}^{\!\top}\mathbf{X})^{-1}.

The degrees of freedom $n-p$ arise because $\mathbf{e}=(\mathbf{I}-\mathbf{H})\mathbf{y}$ lies in the orthogonal complement of a $p$ -dimensional subspace, which has dimension $n-p$ ; the unbiasedness of $s^{2}$ and the $t$ distribution then follow as in the simple case.∎

Pitfall. In multiple regression, $\beta_j$ is the change in $E(Y)$ per unit change in $x_j$ with all other predictors held fixed. That is a different quantity from the simple-regression slope of $y$ on $x_j$ alone, and the two can have opposite signs.

The classic instance: regressing income on years of education gives a positive slope; adding occupation to the model can shrink or reverse it, because education's effect operates partly through occupation, and holding occupation fixed removes that pathway. Neither coefficient is wrong; they answer different questions. Reporting a multiple-regression coefficient as "the effect of $x_j$ " without naming what is being held fixed is ambiguous at best.

Example 7.32 (A two-predictor fit read correctly). A model of house price (thousands) on floor area ( $x_1$ , in $100\,\text{m}^2$ ) and age ( $x_2$ , years) gives $\hat y = 120 + 85x_1 - 1.4x_2$ with standard errors $9.2$ and $0.35$ , from $n = 60$ houses. Interpret, and test each coefficient at $\alpha=0.05$ with $t_{0.025,57}=2.002$ .

Solution.

Interpretation. Among houses of the same age, each additional $100\,\text{m}^2$ is associated with $\$85{,}000$ more. Among houses of the same size, each additional year of age is associated with $\$1{,}400$ less.
Degrees of freedom: $n-p = 60-3 = 57$ , since $p=3$ parameters were fitted.
Area: $t = 85/9.2 = 9.24$ , far beyond $2.002$ — strongly significant.
Age: $t = -1.4/0.35 = -4.00$ , and $|-4.00| > 2.002$ — significant.
Confidence interval for the area coefficient: $85 \pm 2.002(9.2) = 85\pm 18.4 = (66.6,\ 103.4)$ . The association is clearly positive but its magnitude is pinned down only to within about $\pm 22\%$ .
Sanity check on the intercept: $\hat\beta_0 = 120$ is the predicted price of a brand-new house of zero floor area, which does not exist. An intercept is frequently meaningless in isolation and is not evidence of anything wrong with the model — it is a fitted constant, not a prediction at a real design point.

□

Example 7.33 (The normal equations by hand). For the running data $x = 1,2,3,4,5$ , $y = 2,4,5,4,5$ , write $\mathbf{X}^{\!\top}\mathbf{X}$ and $\mathbf{X}^{\!\top}\mathbf{y}$ and solve the normal equations, confirming the scalar answer.

Solution.

The design matrix has a column of ones and the $x$ column, so

\mathbf{X}^{\!\top}\mathbf{X} = \begin{pmatrix} n & \sum x_i \\ \sum x_i & \sum x_i^2 \end{pmatrix} = \begin{pmatrix} 5 & 15 \\ 15 & 55\end{pmatrix}, \qquad \mathbf{X}^{\!\top}\mathbf{y} = \begin{pmatrix} \sum y_i \\ \sum x_iy_i\end{pmatrix} = \begin{pmatrix} 20 \\ 66 \end{pmatrix},

using $\sum x_i^{2} = 1+4+9+16+25 = 55$ and $\sum x_iy_i = 2+8+15+16+25 = 66$ . 2. The determinant is $5(55)-15^2 = 275-225 = 50$ , so

(\mathbf{X}^{\!\top}\mathbf{X})^{-1} = \frac{1}{50}\begin{pmatrix} 55 & -15\\ -15 & 5\end{pmatrix}.

Therefore

\hat{\boldsymbol\beta} = \frac{1}{50}\begin{pmatrix} 55 & -15\\ -15 & 5\end{pmatrix}\begin{pmatrix}20\\66\end{pmatrix} = \frac{1}{50}\begin{pmatrix} 1100-990 \\ -300+330\end{pmatrix} = \frac{1}{50}\begin{pmatrix}110\\30\end{pmatrix} = \begin{pmatrix}2.2\\0.6\end{pmatrix}.

This is exactly $b_0 = 2.2$ , $b_1 = 0.6$ from the scalar computation ✓.
Sanity check on the determinant: $n\sum x_i^{2} - (\sum x_i)^2 = nS_{xx} = 5\times 10 = 50$ ✓, which is the same quantity that appeared in the Hessian in the proof of the theorem The least-squares estimates. It is non-zero exactly when the $x$ 's are not all identical — the full-rank condition in this simplest case.
Standard errors follow from the same inverse: $\operatorname{Var}(\hat\beta_1) = \sigma^{2}\times 5/50 = \sigma^{2}/10 = \sigma^{2}/S_{xx}$ ✓, matching the corollary Precision of the slope.

□

Exercises for §7.6 →

7.7When predictors carry the same information

The matrix formula $\operatorname{Var}(\hat{\boldsymbol\beta}) = \sigma^{2}(\mathbf{X}^{\!\top}\mathbf{X})^{-1}$ warns of a specific failure mode: if $\mathbf{X}^{\!\top}\mathbf{X}$ is near-singular, its inverse has enormous entries and the coefficients are estimated terribly.

Definition 7.34 (Multicollinearity and the variance inflation factor). Multicollinearity is strong linear association among the predictors. Its effect on predictor $j$ is measured by the variance inflation factor

\text{VIF}_j = \frac{1}{1-R_j^2},

where $R_j^{2}$ is the coefficient of determination from regressing $x_j$ on all the other predictors.

Theorem 7.35 (Variance inflation). The variance of the $j$ -th coefficient estimate is

\operatorname{Var}(\hat\beta_j) = \frac{\sigma^2}{S_{jj}}\cdot\frac{1}{1-R_j^2} = \frac{\sigma^2}{S_{jj}}\,\text{VIF}_j,

where $S_{jj}=\sum_i(x_{ij}-\bar x_j)^2$ . So the variance is that of a simple regression on $x_j$ alone, multiplied by the inflation factor.

Proof. We quote the standard identity, whose derivation uses the partitioned-inverse formula for $(\mathbf{X}^{\!\top}\mathbf{X})^{-1}$ . The content is readable directly: regress $x_j$ on the other predictors and let $\mathbf{r}_j$ be the residual vector, the part of $x_j$ not explained by the others. Then $\hat\beta_j$ depends on the data only through $\mathbf{r}_j$ , with

\operatorname{Var}(\hat\beta_j) = \frac{\sigma^2}{\|\mathbf{r}_j\|^2}, \qquad \|\mathbf{r}_j\|^2 = S_{jj}(1-R_j^2),

the second equality being the definition of $R_j^{2}$ applied to that auxiliary regression. Substituting gives the claim.∎

That proof says exactly what is going on, and it is more useful than the formula. A coefficient is estimated from the part of its predictor that the other predictors cannot account for. If $x_j$ is nearly a linear combination of the others, that leftover part is tiny, and the coefficient is being estimated from almost no information.

Remark (What multicollinearity does and does not break). It inflates standard errors, so individual coefficients become insignificant and unstable — dropping one observation can swing them wildly, and two highly correlated predictors often produce large coefficients of opposite sign that nearly cancel.

It does not bias the estimates, and it does not harm prediction within the range of the observed data: the fitted values and $R^{2}$ are unaffected, because the column space of $\mathbf{X}$ is unchanged. This is the resolution of a common puzzle — a model with a highly significant $F$ and not one significant coefficient. The predictors jointly explain a great deal; the data simply cannot apportion the credit.

So the remedy depends on the goal. For prediction, do nothing. For interpreting individual coefficients, drop one of the redundant predictors, combine them into an index, or collect data that breaks the association.

Example 7.36 (Reading VIFs). A model has $R_j^{2} = 0.90$ for one predictor and $0.20$ for another. Compare their variance inflation factors and the effect on the standard errors.

Solution.

$\text{VIF} = 1/(1-0.90) = 10$ for the first; $\text{VIF} = 1/(1-0.20) = 1.25$ for the second.
Standard errors scale as $\sqrt{\text{VIF}}$ , so the first coefficient's standard error is $\sqrt{10} = 3.16$ times what it would be with an uncorrelated predictor; the second's is $\sqrt{1.25}=1.12$ times.
To compensate, the first predictor would need about $10$ times as many observations to reach the same precision, since variance falls as $1/n$ .
Sanity check on the usual thresholds: rules of thumb flag $\text{VIF} > 5$ or $> 10$ . Those correspond to $R_j^{2} > 0.8$ and $R_j^{2} > 0.9$ — very strong association among predictors, and the thresholds are conventions rather than results.
Note what is not implied: the first predictor may still belong in the model. A large VIF is a statement about precision, not about relevance.

□

Exercises for §7.7 →

7.8Choosing a model

Adding a predictor can never increase $SSE$ — the larger model contains the smaller one as the special case where the new coefficient is zero, so its minimised sum of squares is at least as small. $R^{2}$ therefore rises with every predictor added, including pure noise, and cannot be used to compare models of different sizes.

Theorem 7.37 (Nested model comparison). Let a full model with $p$ parameters contain a reduced model with $q < p$ parameters as a special case. To test whether the $p-q$ extra coefficients are all zero,

F = \frac{(SSE_{\text{red}} - SSE_{\text{full}})/(p-q)}{SSE_{\text{full}}/(n-p)} \sim F_{\,p-q,\;n-p} \quad\text{under } H_0,

rejecting for large $F$ .

Proof. Both residual sums of squares are squared lengths of projections onto orthogonal complements of nested subspaces, so $SSE_{\text{red}} \ge SSE_{\text{full}}$ and the difference is the squared length of the projection onto the $(p-q)$ -dimensional part of the full column space orthogonal to the reduced one — by the theorem Least squares is orthogonal projection applied twice.

Under $H_0$ that component consists of noise alone, so $(SSE_{\text{red}}-SSE_{\text{full}})/\sigma^{2} \sim \chi^{2}_{p-q}$ , independently of $SSE_{\text{full}}/\sigma^{2}\sim\chi^{2}_{n-p}$ since the two live in orthogonal subspaces. The ratio of independent chi-squares each over its degrees of freedom is $F$ , with $\sigma^{2}$ cancelling.∎

Taking the reduced model to be intercept-only recovers the overall $F$ -test of the whole regression, and taking $p-q=1$ recovers the $t$ -test of a single coefficient, since $F_{1,\nu}=t_\nu^{2}$ . It is the same machinery as the ANOVA $F$ , which the previous chapter showed was this test all along.

Example 7.38 (Do three predictors earn their place?). A model with $n=40$ has $SSE = 120$ with $p=6$ parameters. Dropping three predictors gives $SSE = 165$ with $q=3$ . Test at $\alpha=0.05$ , given $F_{0.05,3,34}=2.883$ .

Solution.

The reduction in $SSE$ from the three predictors is $165-120 = 45$ , on $p-q = 3$ degrees of freedom, so the numerator is $45/3 = 15$ .
The denominator is $SSE_{\text{full}}/(n-p) = 120/34 = 3.529$ .
$F = 15/3.529 = 4.250$ . Since $4.250 > 2.883$ , reject: at least one of the three carries information beyond the reduced model.
Sanity check on what this does not say: the joint test does not identify which of the three matters, exactly as a significant one-way ANOVA does not identify which groups differ. Examining the three individual $t$ statistics afterwards is three more comparisons.
Note the contrast with the individual tests: under multicollinearity all three coefficients could be individually insignificant while this joint test is highly significant, because the three together explain a lot even when no one of them is separable from the others.

□

Definition 7.39 (Adjusted R-squared).

R^2_{\text{adj}} = 1 - \frac{SSE/(n-p)}{SST/(n-1)} = 1 - (1-R^2)\frac{n-1}{n-p}.

Proposition 7.40 (Adjusted R-squared penalises useless predictors). Adding a predictor increases $R^{2}_{\text{adj}}$ only if it reduces $SSE$ by more than the factor by which it reduces the remaining degrees of freedom. A predictor whose $t$ statistic has $|t| < 1$ always decreases $R^{2}_{\text{adj}}$ .

Proof. $R^{2}_{\text{adj}}$ increases exactly when $SSE/(n-p)$ — the estimate $s^{2}$ — decreases, since $SST/(n-1)$ is fixed. Adding a predictor changes the numerator from $SSE_{\text{small}}$ to $SSE_{\text{small}} - \Delta$ and the denominator from $n-p$ to $n-p-1$ , so $s^{2}$ falls exactly when

\frac{SSE - \Delta}{n-p-1} < \frac{SSE}{n-p} \iff \Delta > \frac{SSE}{n-p} = s^2_{\text{small}} .

The reduction $\Delta$ in $SSE$ from adding one predictor equals $t^{2} s^{2}_{\text{large}}$ for that predictor's $t$ statistic, so the condition is essentially $t^{2} > 1$ .∎

Remark (Information criteria and regularisation). Two other approaches trade fit against complexity more formally. AIC $= n\log(SSE/n) + 2p$ and BIC $= n\log(SSE/n) + p\log n$ both add a penalty per parameter, with BIC's penalty larger for $n > 7$ , so BIC selects smaller models. Lower is better for both, and neither is a hypothesis test — they rank models rather than accepting or rejecting.

A different strategy shrinks the coefficients instead of dropping predictors. Ridge regression minimises $\|\mathbf{y}-\mathbf{X}\mathbf{b}\|^2 + \lambda\|\mathbf{b}\|^2$ , which has the closed form $\hat{\boldsymbol\beta}_{\text{ridge}} = (\mathbf{X}^{\!\top}\mathbf{X}+\lambda\mathbf{I})^{-1}\mathbf{X}^{\!\top}\mathbf{y}$ — note that adding $\lambda\mathbf{I}$ makes the matrix invertible even when $\mathbf{X}^{\!\top}\mathbf{X}$ is not, which is precisely the multicollinearity cure. Lasso uses $\lambda\sum_j|b_j|$ instead and sets some coefficients exactly to zero, performing selection and fitting at once.

Both are deliberately biased, and by the theorem Bias-variance decomposition that is the point: they accept bias to buy a larger reduction in variance, and can beat least squares on mean squared error despite Gauss-Markov — which constrains only unbiased estimators.

Pitfall. Selecting predictors by looking at $p$ -values, then reporting the final model's $p$ -values as though the search had not happened, is the multiplicity error of the testing chapter in its most common form. Stepwise selection over $20$ candidate predictors examines far more than $20$ hypotheses, and the surviving coefficients are biased away from zero because they survived.

If the model was chosen using the data, its reported standard errors and intervals are not valid for that data. The honest options are to select on one part of the data and assess on another, to use cross-validation, or to report the selection procedure and treat the result as exploratory.

Example 7.41 (Adjusted R-squared in use). A model with $n=50$ and $p=4$ has $R^{2} = 0.62$ . A fifth predictor raises $R^{2}$ to $0.63$ . Compute both adjusted values and decide.

Solution.

Four parameters: $R^{2}_{\text{adj}} = 1 - (1-0.62)\dfrac{49}{46} = 1 - 0.38\times 1.06522 = 1-0.40478 = 0.5952$ .
Five parameters: $R^{2}_{\text{adj}} = 1 - (1-0.63)\dfrac{49}{45} = 1 - 0.37\times 1.08889 = 1-0.40289 = 0.5971$ .
The adjusted value rises very slightly, from $0.5952$ to $0.5971$ , so the predictor just about pays for itself.
Sanity check via the $t$ criterion: by the proposition Adjusted R-squared penalises useless predictors, the increase means the new predictor's $|t|$ exceeds $1$ — which is a very weak bar, far below the $|t| > 2.0$ needed for significance. So "improves adjusted $R^{2}$ " is not the same as "is worth keeping".
The gain of $0.0019$ in adjusted $R^{2}$ is not a reason to prefer the larger model. With this little to separate them, the simpler model is the better default.

□

Exercises for §7.8 →

7.9Extrapolation, and the limits of what a fit can say

Definition 7.42 (Extrapolation). Extrapolation is prediction at a predictor value outside the range spanned by the observed data.

The intervals of the theorem Confidence and prediction intervals already price part of the cost: the term $(x_0-\bar x)^2/S_{xx}$ grows quadratically as $x_0$ leaves the data, so both bands flare outward. But that widening assumes the model still holds, and the real danger is that it does not. A relationship linear over the observed range may bend, saturate, or reverse outside it, and the data contain no information whatever about which.

Pitfall. The most consequential limitation is that a regression coefficient is not a causal effect. Least squares fits a conditional mean; nothing in the derivation licenses a claim about what would happen under intervention.

Three mechanisms break the inference, and none is detectable from the fit:

Confounding. A variable associated with both $x$ and $y$ creates an association between them with no causal path. Ice cream sales predict drownings; temperature drives both.
Reverse causation. The regression is symmetric in a sense the causal claim is not — fitting $x$ on $y$ gives a slope $r\,s_x/s_y$ , and both fits describe the same data equally well.
Selection. If inclusion in the sample depends on both variables, an association appears among those selected even when none exists in the population.

Adding controls addresses only confounders you measured and thought of, and adding the wrong control — one on the causal pathway, or one caused by both variables — can create bias rather than remove it. Design is what licenses causal claims, which is the subject of the next chapter.

Example 7.43 (The same data, two regressions). For the running example, fit $x$ on $y$ as well as $y$ on $x$ , and compare.

Solution.

$y$ on $x$ : slope $S_{xy}/S_{xx} = 6/10 = 0.6$ , giving $\hat y = 2.2+0.6x$ .
$x$ on $y$ : slope $S_{xy}/S_{yy} = 6/6 = 1.0$ , and intercept $\bar x - 1.0\bar y = 3-4 = -1$ , giving $\hat x = -1 + 1.0y$ .
If the first line could simply be inverted, it would give $x = (y-2.2)/0.6 = -3.667 + 1.667y$ — a slope of $1.667$ , not $1.0$ . The two fits are different lines.
The product of the two slopes is $0.6\times 1.0 = 0.6 = r^{2}$ , which is exact and general: $b_{y|x}b_{x|y} = S_{xy}^2/(S_{xx}S_{yy}) = r^{2}$ . The lines coincide only when $r^{2}=1$ .
Sanity check on what this means: each regression minimises errors in a different direction — vertical for $y$ on $x$ , horizontal for $x$ on $y$ — so asking "which variable causes which" cannot be settled by fitting, since both fits are equally valid descriptions. The asymmetry of a causal claim has no counterpart in the algebra.

□

Exercises for §7.9 →

Common Mistakes

**Reading $r$ as a measure of association in general.** It measures *linear* association; $y=x^{2}$ over a symmetric range gives $r=0$ .
Quoting a correlation or a fit without plotting the data. Anscombe's quartet shares every summary statistic across four data sets that demand four different models.
**Treating $R^{2}$ as a verdict on the model.** It says how much variation the fit accounts for, not whether the form is right, and it rises with every predictor added.
Confusing a confidence interval for the mean response with a prediction interval. The second carries an extra $\sigma^{2}$ that never shrinks with more data.
Reading a multiple-regression coefficient without saying what is held fixed. It is a partial effect, and can differ in sign from the simple-regression slope.
Concluding a predictor is useless because its coefficient is insignificant under multicollinearity. Inflated standard errors say the data cannot apportion credit, not that the predictor is irrelevant.
**Reporting $p$ -values from a model selected using the same data.** The surviving coefficients are biased away from zero precisely because they survived.
Extrapolating. The bands widen quadratically outside the data, and even that assumes the model still holds, which the data cannot confirm.
Calling a coefficient an effect. Confounding, reverse causation and selection all produce coefficients, and none is visible in the residual plots.

About the practice questions. They check that you can carry out this chapter's computations correctly, and each one is graded on a single answer. They are not proof exercises: working through them confirms the mechanics, not that you could prove the results yourself. For that, re-read the statements above and try to reconstruct their proofs with the page closed.

← Design of Experiments Logistic Regression →

Helpful?