Statistics · Solutions

Correlation and Regression

12 questions · about 101 minutes

Work this sheet on paper, showing every step — the method is what is being checked, and the solutions are written so you can follow your own route rather than just compare a final number. Name every result you use (“by Gauss–Markov”, “by the decomposition of variation”, “from the normal equations”) and say which of the LINE conditions each inferential step relies on. The arithmetic has been chosen so that no calculator is needed: leave surds as surds and use only the approximations supplied in the questions. Full worked solutions are on a separate page; each question also has its own reveal, so you can check one answer without seeing the rest.

1.Computation
Four paired observations are recorded: $(2, 1), (4, 2), (6, 4), (8, 5)$ .
1. (a)Compute $S_{xx}$ , $S_{yy}$ , $S_{x y}$ and the sample correlation $r$ exactly, leaving any surd as a surd. Check your value against the theorem that bounds $r$ .
2. (b)The $x$ values are re-expressed as $x^{'} = 3 x + 1$ and the $y$ values as $y^{'} = 4 - y$ . Without recomputing any sum, state the new correlation $r^{'}$ and justify it by naming the result you are using. Then confirm it by computing $S_{x^{'} y^{'}}$ , $S_{x^{'} x^{'}}$ and $S_{y^{'} y^{'}}$ from the originals.
3. (c)Now take the five points $x = - 2, - 1, 0, 1, 2$ with $y = x^{2}$ . Compute $r$ , and say precisely what this example shows about what the correlation does and does not measure.
Solution
(a) Centre, then sum. $\overset{x}{ˉ} = 20/4 = 5$ and $\overset{y}{ˉ} = 12/4 = 3$ . The deviations are $u = - 3, - 1, 1, 3$ and $v = - 2, - 1, 1, 2$ , so
$S_{xx} = 9 + 1 + 1 + 9 = 20, S_{yy} = 4 + 1 + 1 + 4 = 10, S_{x y} = 6 + 1 + 1 + 6 = 14.$
Hence
$r = \frac{S _{x y}}{S _{xx} S _{yy}} = \frac{14}{200} = \frac{14}{10 2} = \frac{7 2}{10} \approx 0.99.$
Sanity check: $r^{2} = 98/100 < 1$ , as the theorem Correlation lies between $- 1$ and $1$ requires, and it is strictly less than $1$ because the points are not collinear — the slope from $(2, 1)$ to $(4, 2)$ is $\frac{1}{2}$ , while from $(4, 2)$ to $(6, 4)$ it is $1$ .
(b) Invoke invariance; recompute nothing. By the proposition Correlation is invariant under increasing linear rescaling, if $x^{'} = a + b x$ and $y^{'} = c + d y$ then $r^{'} = sign (b d) r$ . Here $b = 3$ and $d = - 1$ , so $b d < 0$ and
$r^{'} = - \frac{7 2}{10} .$
The shifts $+ 1$ and $4$ do nothing, because centring removes them; the factor $3$ scales $S_{x y}$ and $S_{xx}$ equally and cancels; only the sign of $d$ survives.
Confirmation: centring gives $x_{i}^{'} - \overset{x}{ˉ}^{'} = 3 (x_{i} - \overset{x}{ˉ})$ and $y_{i}^{'} - \overset{y}{ˉ}^{'} = - (y_{i} - \overset{y}{ˉ})$ , so $S_{x^{'} y^{'}} = 3 \cdot (- 1) \cdot 14 = - 42$ , $S_{x^{'} x^{'}} = 9 \times 20 = 180$ and $S_{y^{'} y^{'}} = 1 \times 10 = 10$ . Then
$r^{'} = \frac{- 42}{1800} = \frac{- 42}{30 2} = - \frac{7}{5 2} = - \frac{7 2}{10} . ✓$
(c) A perfect relationship with zero correlation. $\overset{x}{ˉ} = 0$ and $\overset{y}{ˉ} = (4 + 1 + 0 + 1 + 4) /5 = 2$ . The $y$ -deviations are $v = 2, - 1, - 2, - 1, 2$ , and since $\overset{x}{ˉ} = 0$ the $x$ -deviations are the $x$ values themselves:
$S_{x y} = (- 2) (2) + (- 1) (- 1) + (0) (- 2) + (1) (- 1) + (2) (2) = - 4 + 1 + 0 - 1 + 4 = 0.$
So $r = 0$ exactly, even though $y$ is determined by $x$ without any noise at all. (For the record $S_{xx} = 10$ and $S_{yy} = 4 + 1 + 4 + 1 + 4 = 14$ , both positive, so $r$ is well defined.)
The lesson: $r$ measures linear association only. Every product $u_{i} v_{i}$ on the left of $0$ is cancelled by its mirror image on the right, because the parabola rises on one side exactly as it falls on the other. A correlation of zero is compatible with a deterministic relationship, which is why a correlation quoted without a scatterplot has not had its meaning checked.
2.Computation
For the four points $(2, 1), (4, 2), (6, 4), (8, 5)$ you may take $\overset{x}{ˉ} = 5$ , $\overset{y}{ˉ} = 3$ , $S_{xx} = 20$ , $S_{yy} = 10$ , $S_{x y} = 14$ and $r = 72 /10$ as given.
1. (a)Find the least-squares line $\overset{y}{^} = b_{0} + b_{1} x$ , naming the theorem that supplies the formulas, and check that the line passes through the point of means.
2. (b)Compute the four fitted values and the four residuals, and verify both identities $\sum_{i} e_{i} = 0$ and $\sum_{i} x_{i} e_{i} = 0$ numerically.
3. (c)Verify the relation $b_{1} = r s_{y} / s_{x}$ for this data, and use it to explain what is meant by “regression to the mean” for a point one standard deviation above $\overset{x}{ˉ}$ .
Solution
(a) The least-squares estimates. By the theorem The least-squares estimates, the minimiser of $\sum_{i} (y_{i} - b_{0} - b_{1} x_{i})^{2}$ is
$b_{1} = \frac{S _{x y}}{S _{xx}} = \frac{14}{20} = \frac{7}{10}, b_{0} = \overset{y}{ˉ} - b_{1} \overset{x}{ˉ} = 3 - \frac{7}{10} \cdot 5 = 3 - \frac{7}{2} = - \frac{1}{2} .$
The line is $\overset{y}{^} = - 0.5 + 0.7 x$ . At $x = \overset{x}{ˉ} = 5$ it gives $- 0.5 + 3.5 = 3 = \overset{y}{ˉ}$ , so it passes through $(\overset{x}{ˉ}, \overset{y}{ˉ})$ — which is exactly what the first normal equation says.
(b) Fitted values and residuals.
$\overset{y}{^} = 0.9, 2.3, 3.7, 5.1 at x = 2, 4, 6, 8,$
so the residuals $e_{i} = y_{i} - \overset{y}{^}_{i}$ are
$e = 1 - 0.9, 2 - 2.3, 4 - 3.7, 5 - 5.1 = 0.1, - 0.3, 0.3, - 0.1.$
Sum: $0.1 - 0.3 + 0.3 - 0.1 = 0$ . $✓$ Weighted sum: $2 (0.1) + 4 (- 0.3) + 6 (0.3) + 8 (- 0.1) = 0.2 - 1.2 + 1.8 - 0.8 = 0$ . $✓$ Both hold by the proposition Two properties of least-squares residuals — they are the two normal equations rewritten, so they hold for every data set, not just this one.
(c) Slope equals correlation in standard units. $s_{y} / s_{x} = S_{yy} / S_{xx}$ , since both standard deviations carry the same divisor $n - 1$ , which cancels. So
$r \frac{s _{y}}{s _{x}} = \frac{7 2}{10} \cdot \frac{10}{20} = \frac{7 2}{10} \cdot \frac{1}{2} = \frac{7}{10} = b_{1} . ✓$
This is the corollary Slope, correlation and the regression effect. Read it in standard units: a point at $\overset{x}{ˉ} + s_{x}$ is predicted at $\overset{y}{ˉ} + r s_{y}$ , that is $r \approx 0.99$ standard deviations above $\overset{y}{ˉ}$ — closer to its mean, in relative terms, than the predictor was to its own. With $r$ this close to $1$ the pull is slight; with $r = 0.5$ the prediction would sit only half a standard deviation up. Nothing is acting on anything: regression to the mean is what imperfect correlation looks like when you predict.
3.Proof
Let $b_{0}, b_{1}$ be the least-squares estimates for data $(x_{i}, y_{i})$ , $i = 1, \dots, n$ , with $S_{xx} > 0$ , and let $\overset{y}{^}_{i} = b_{0} + b_{1} x_{i}$ and $e_{i} = y_{i} - \overset{y}{^}_{i}$ . Work only from the fact that $(b_{0}, b_{1})$ minimises $Q (b_{0}, b_{1}) = \sum_{i} (y_{i} - b_{0} - b_{1} x_{i})^{2}$ .
1. (a)Prove that $\sum_{i} e_{i} = 0$ and $\sum_{i} x_{i} e_{i} = 0$ .
2. (b)Deduce that $\sum_{i} \overset{y}{^}_{i} e_{i} = 0$ .
3. (c)Deduce that the sample correlation between the $x_{i}$ and the $e_{i}$ is exactly zero whenever the residuals are not all zero, and explain what this means for a plot of residuals against $x$ .
Solution
(a) The normal equations. $Q$ is a differentiable function of $(b_{0}, b_{1})$ and the minimum is attained (it is a positive-definite quadratic when $S_{xx} > 0$ , as the theorem The least-squares estimates shows), so both partial derivatives vanish there:
$\frac{\partial Q}{\partial b _{0}} = - 2 i \sum (y_{i} - b_{0} - b_{1} x_{i}) = - 2 i \sum e_{i} = 0,$
$\frac{\partial Q}{\partial b _{1}} = - 2 i \sum x_{i} (y_{i} - b_{0} - b_{1} x_{i}) = - 2 i \sum x_{i} e_{i} = 0.$
Dividing by $- 2$ gives $\sum_{i} e_{i} = 0$ and $\sum_{i} x_{i} e_{i} = 0$ . $■$
(b) Fitted values are a combination of $1$ and $x$ . Since $\overset{y}{^}_{i} = b_{0} + b_{1} x_{i}$ ,
$i \sum \overset{y}{^}_{i} e_{i} = b_{0} i \sum e_{i} + b_{1} i \sum x_{i} e_{i} = b_{0} \cdot 0 + b_{1} \cdot 0 = 0. ■$
Geometrically: the residual vector is orthogonal to the column of ones and to the column of $x$ 's, hence to everything in their span, and the fitted vector lies in that span.
(c) Residuals are uncorrelated with the predictor. By (a) the mean residual is $\overset{e}{ˉ} = \frac{1}{n} \sum_{i} e_{i} = 0$ . Therefore
$S_{x e} = i \sum (x_{i} - \overset{x}{ˉ}) (e_{i} - \overset{e}{ˉ}) = i \sum (x_{i} - \overset{x}{ˉ}) e_{i} = i \sum x_{i} e_{i} - \overset{x}{ˉ} i \sum e_{i} = 0 - 0 = 0.$
If the residuals are not all zero then $S_{ee} > 0$ , and $S_{xx} > 0$ by hypothesis, so $r_{x e} = S_{x e} / S_{xx} S_{ee} = 0$ . $■$
Consequence: a plot of $e$ against $x$ can never show a linear trend, whatever the data — a least-squares line through that plot would have slope $S_{x e} / S_{xx} = 0$ . That is a property of how the line was chosen, not evidence about the data. What the plot can show is anything non-linear: curvature, or a spread that changes with $x$ . Those are the two things the fitting procedure did not remove, and they are exactly what the residual plot is used to diagnose.
4.Computation
The line $\overset{y}{^} = - 0.5 + 0.7 x$ was fitted to $(2, 1), (4, 2), (6, 4), (8, 5)$ , with residuals $0.1, - 0.3, 0.3, - 0.1$ , and $S_{xx} = 20$ , $S_{yy} = 10$ , $S_{x y} = 14$ , $r = 72 /10$ .
1. (a)Compute $SSR$ from the identity $SSR = b_{1} S_{x y}$ , compute $SSE$ directly from the residuals, and verify the decomposition $SST = SSR + SSE$ . State why the cross term in that decomposition vanishes.
2. (b)Compute $R^{2}$ and confirm that it equals $r^{2}$ .
3. (c)Compute $s^{2}$ and $s$ exactly, say in one sentence what $s$ measures, and explain why $R^{2} = 0.98$ does not by itself establish that a straight line is the right model.
Solution
(a) The three sums of squares. By the proposition For simple regression, R-squared is r squared, $SSR = b_{1} S_{x y} = 0.7 \times 14 = 9.8$ ; as a check, $b_{1}^{2} S_{xx} = 0.49 \times 20 = 9.8$ as well. Directly from the residuals,
$SSE = 0.01 + 0.09 + 0.09 + 0.01 = 0.20.$
With $SST = S_{yy} = 10$ we get $SSR + SSE = 9.8 + 0.2 = 10 = SST$ . $✓$
The cross term $2 \sum_{i} (\overset{y}{^}_{i} - \overset{y}{ˉ}) (y_{i} - \overset{y}{^}_{i})$ vanishes because $\overset{y}{^}_{i} - \overset{y}{ˉ} = b_{1} (x_{i} - \overset{x}{ˉ})$ , so it equals $2 b_{1} (\sum_{i} x_{i} e_{i} - \overset{x}{ˉ} \sum_{i} e_{i}) = 2 b_{1} (0 - 0) = 0$ by the proposition Two properties of least-squares residuals — this is the theorem Decomposition of variation in regression.
(b) Coefficient of determination.
$R^{2} = \frac{SSR}{SST} = \frac{9.8}{10} = 0.98, r^{2} = (\frac{7 2}{10})^{2} = \frac{49 \times 2}{100} = 0.98. ✓$
The line accounts for $98%$ of the variation in $y$ .
(c) Residual standard error. Two parameters were estimated, so the divisor is $n - 2 = 2$ :
$s^{2} = \frac{SSE}{n - 2} = \frac{0.2}{2} = 0.1, s = \frac{1}{10} = \frac{10}{10} \approx 0.32.$
$s$ estimates $σ$ , the standard deviation of an observation about the true line — roughly, how far a typical point sits from the fitted line in the units of $y$ . Against $\overset{y}{ˉ} = 3$ that is a small scatter, consistent with the high $R^{2}$ .
But $R^{2}$ is computed from three sums of squares and is blind to the pattern of the residuals. Anscombe's quartet has four data sets with identical $r$ , identical fitted line and identical $R^{2}$ , of which one is curved, one is driven by a single high-leverage point and one contains an outlier. Here the residuals $0.1, - 0.3, 0.3, - 0.1$ alternate in sign with no trend, which is reassuring — but that reassurance came from looking at the residuals, not from the $0.98$ .
5.Sketch
An experimenter measures a response at $x = 0, 1, 2, 3, 4$ and records $y = 0, 1, 4, 9, 16$ . (The response is in fact $y = x^{2}$ exactly, but the experimenter does not know this and fits a straight line.)
1. (a)Fit the least-squares line.
2. (b)Compute the five fitted values and residuals, then $SSE$ , $SST$ and $R^{2}$ as an exact fraction.
3. (c)Sketch the scatterplot with the fitted line drawn on it, and beneath it the plot of residuals against fitted values. Every feature you draw must be labelled with a number from (a) or (b). Name the LINE condition the second plot shows to be violated, and explain why the $R^{2}$ of (b) failed to reveal it.
Solution
(a) Fit the line. $\overset{x}{ˉ} = 2$ and $\overset{y}{ˉ} = 30/5 = 6$ . Deviations: $u = - 2, - 1, 0, 1, 2$ and $v = - 6, - 5, - 2, 3, 10$ . Then
$S_{xx} = 4 + 1 + 0 + 1 + 4 = 10, S_{x y} = 12 + 5 + 0 + 3 + 20 = 40,$
so $b_{1} = 40/10 = 4$ and $b_{0} = 6 - 4 \cdot 2 = - 2$ . The fitted line is $\overset{y}{^} = - 2 + 4 x$ .
(b) Residuals and the sums of squares. Fitted values at $x = 0, 1, 2, 3, 4$ are $- 2, 2, 6, 10, 14$ , so the residuals are
$e = 0 - (- 2), 1 - 2, 4 - 6, 9 - 10, 16 - 14 = 2, - 1, - 2, - 1, 2.$
Check: $\sum e_{i} = 0$ and $\sum x_{i} e_{i} = 0 - 1 - 4 - 3 + 8 = 0$ . $✓$
$SSE = 4 + 1 + 4 + 1 + 4 = 14, SST = S_{yy} = 36 + 25 + 4 + 9 + 100 = 174,$
and $SSR = b_{1} S_{x y} = 160$ , with $160 + 14 = 174$ . $✓$ Hence
$R^{2} = \frac{160}{174} = \frac{80}{87} \approx 0.92.$
(c) The two pictures. Scatterplot. Mark the five points $(0, 0), (1, 1), (2, 4), (3, 9), (4, 16)$ ; they rise ever more steeply. Draw the line $y = - 2 + 4 x$ through $(0, - 2)$ and $(4, 14)$ ; it passes through the point of means $(2, 6)$ . The line lies below the data at both ends — at $x = 0$ the point is $2$ above it, at $x = 4$ it is $2$ above — and above the data in the middle, by $1, 2, 1$ at $x = 1, 2, 3$ . The line crosses the curve where $x^{2} = 4 x - 2$ , that is at $x = 2 \pm 2 \approx 0.6$ and $3.4$ ; mark those two crossings.
Residual plot. Horizontal axis: fitted value, from $- 2$ to $14$ ; vertical axis: residual, from $- 2$ to $2$ , with a dashed line at $0$ . Plot $(- 2, 2), (2, - 1), (6, - 2), (10, - 1), (14, 2)$ . The points trace a clear U: positive at both ends, negative in the middle, symmetric about the centre. This is not a random band about zero.
Which condition fails. Linearity. $E (Y ∣ x)$ is $x^{2}$ , not $β_{0} + β_{1} x$ , so the line is systematically too low at the ends and too high in the middle, and the residuals inherit that curvature. (Independence, normality and equal variance are not what this plot indicts — there is no noise at all.)
Why $R^{2}$ did not see it. $R^{2} = 1 - SSE / SST$ depends on the residuals only through the single number $\sum e_{i}^{2} = 14$ . Reorder the same five residuals to $2, - 2, 2, - 1, - 1$ and $R^{2}$ is unchanged, though the U has vanished. A fit accounting for $92%$ of the variation sounds excellent, and the line is a decent summary of the trend — but the model is wrong in a way that would ruin every prediction at the ends, and only the residual plot says so. That is the pitfall of Anscombe's quartet made concrete.
6.Proof
Assume $Y_{i} = β_{0} + β_{1} x_{i} + ε_{i}$ with the errors independent, $E (ε_{i}) = 0$ and $Var (ε_{i}) = σ^{2}$ for every $i$ . Normality is not assumed. Write $w_{i} = (x_{i} - \overset{x}{ˉ}) / S_{xx}$ .
1. (a)Show that $b_{1} = \sum_{i} w_{i} Y_{i}$ , and prove the three identities $\sum_{i} w_{i} = 0$ , $\sum_{i} w_{i} x_{i} = 1$ and $\sum_{i} w_{i}^{2} = 1/ S_{xx}$ .
2. (b)Deduce that $E (b_{1}) = β_{1}$ and $Var (b_{1}) = σ^{2} / S_{xx}$ , stating at each step which assumption is being used.
3. (c)Four observations are to be taken on the interval $[2, 8]$ . Design A uses $x = 2, 4, 6, 8$ ; design B uses $x = 2, 2, 8, 8$ . Compute $Var (b_{1})$ under each, say which is more precise and by what factor, and explain what design B is incapable of detecting.
Solution
(a) The slope is linear in the responses. Since $\sum_{i} (x_{i} - \overset{x}{ˉ}) = 0$ , the constant $\overset{ˉ}{Y}$ drops out of the covariance sum:
$S_{x Y} = i \sum (x_{i} - \overset{x}{ˉ}) (Y_{i} - \overset{ˉ}{Y}) = i \sum (x_{i} - \overset{x}{ˉ}) Y_{i} - \overset{ˉ}{Y} i \sum (x_{i} - \overset{x}{ˉ}) = i \sum (x_{i} - \overset{x}{ˉ}) Y_{i},$
so $b_{1} = S_{x Y} / S_{xx} = \sum_{i} w_{i} Y_{i}$ . For the identities:
$i \sum w_{i} = \frac{1}{S _{xx}} i \sum (x_{i} - \overset{x}{ˉ}) = 0;$
$i \sum w_{i} x_{i} = \frac{1}{S _{xx}} i \sum (x_{i} - \overset{x}{ˉ}) [(x_{i} - \overset{x}{ˉ}) + \overset{x}{ˉ}] = \frac{1}{S _{xx}} (S_{xx} + \overset{x}{ˉ} \cdot 0) = 1;$
$i \sum w_{i}^{2} = \frac{1}{S _{xx}^{2}} i \sum (x_{i} - \overset{x}{ˉ})^{2} = \frac{S _{xx}}{S _{xx}^{2}} = \frac{1}{S _{xx}} . ■$
(b) Mean and variance. Linearity of the model and $E (ε_{i}) = 0$ give $E (Y_{i}) = β_{0} + β_{1} x_{i}$ , so by linearity of expectation
$E (b_{1}) = i \sum w_{i} (β_{0} + β_{1} x_{i}) = β_{0} i \sum w_{i} + β_{1} i \sum w_{i} x_{i} = β_{0} \cdot 0 + β_{1} \cdot 1 = β_{1} .$
For the variance, independence kills every cross term, so the variance of the sum is the sum of the variances: $Var (b_{1}) = \sum_{i} w_{i}^{2} Var (Y_{i})$ . Equal variance makes every $Var (Y_{i}) = σ^{2}$ , hence
$Var (b_{1}) = σ^{2} i \sum w_{i}^{2} = \frac{σ ^{2}}{S _{xx}} . ■$
Normality was never used — this is the unbiasedness and variance part of the theorem Gauss–Markov, and the corollary Precision of the slope. Normality enters only when the $t$ distribution is needed for intervals.
(c) Two designs. Under A, $\overset{x}{ˉ} = 5$ with deviations $\pm 3, \pm 1$ , so $S_{xx} = 9 + 1 + 1 + 9 = 20$ and $Var (b_{1}) = σ^{2} /20$ . Under B, $\overset{x}{ˉ} = 5$ with every deviation $\pm 3$ , so $S_{xx} = 4 \times 9 = 36$ and $Var (b_{1}) = σ^{2} /36$ .
Design B is more precise, by the factor $36/20 = 9/5 = 1.8$ : pushing the $x$ 's to the ends of the interval maximises $S_{xx}$ , which is the denominator in the variance. This is the remark How to make a slope precise — with a trusted linear model, half the points at each extreme is optimal.
What B cannot do is check linearity. With only two distinct $x$ values there are only two group means, and a line has two parameters, so the least-squares line passes exactly through both means. Every residual is then a within-group deviation; the residual-versus-fitted plot has two vertical stacks and no capacity to bend. Had the true relationship been curved between $2$ and $8$ , design B would report a perfect straight line and could never know otherwise. Design A gives up some precision to buy the ability to see the U of the previous question.
7.Computation
For the four points $(2, 1), (4, 2), (6, 4), (8, 5)$ : $\overset{y}{^} = - 0.5 + 0.7 x$ , $n = 4$ , $\overset{x}{ˉ} = 5$ , $S_{xx} = 20$ , $SSR = 9.8$ , $SSE = 0.2$ , $s^{2} = 0.1$ . Assume the full LINE conditions. You may use $t_{0.025, 2} = 4.303$ , $2 \approx 1.414$ and $10 \approx 3.162$ .
1. (a)Compute the standard error of $b_{1}$ exactly, and test $H_{0} : β_{1} = 0$ against $H_{1} : β_{1} \neq = 0$ at $α = 0.05$ . State the sampling distribution you are using and its degrees of freedom.
2. (b)Give a $95%$ confidence interval for $β_{1}$ .
3. (c)At $x_{0} = 5$ give a $95%$ confidence interval for the mean response $E (Y ∣ x_{0})$ and a $95%$ prediction interval for a single new observation. Express both standard errors exactly, identify the term responsible for the difference, and state the exact ratio of the two half-widths.
4. (d)Compute $F = MSR / MSE$ and verify that it equals $t^{2}$ from (a).
Solution
(a) Standard error and test. By the corollary Precision of the slope,
$SE (b_{1}) = \frac{s}{S _{xx}} = \frac{0.1}{20} = \frac{1}{200} = \frac{1}{10 2} = \frac{2}{20} \approx 0.0707.$
By the theorem Sampling distribution of the slope, under $H_{0}$ the statistic $b_{1} / SE (b_{1})$ follows $t_{n - 2} = t_{2}$ . Here
$t = \frac{0.7}{2 /20} = \frac{14}{2} = 72 \approx 9.90.$
Since $9.90 > 4.303$ , reject $H_{0}$ : the predictor carries linear information about the response, even with only two degrees of freedom to spare.
(b) Interval for the slope. $b_{1} \pm t_{0.025, 2} SE (b_{1}) = 0.7 \pm 4.303 \times 0.0707 = 0.7 \pm 0.304$ , so the interval is approximately $(0.40, 1.00)$ . It excludes $0$ , in agreement with (a), and it is wide: two degrees of freedom make $t_{0.025, 2}$ more than twice the familiar $1.96$ .
(c) Two intervals at the centre. $\overset{y}{^}_{0} = - 0.5 + 0.7 \times 5 = 3$ , and $(x_{0} - \overset{x}{ˉ})^{2} = 0$ , so by the theorem Confidence and prediction intervals:
Mean response. $SE = s 1/ n = s /2 = \frac{1}{2 10} = \frac{10}{20} \approx 0.158$ . Margin $4.303 \times 0.158 \approx 0.68$ ; interval $(2.32, 3.68)$ .
New observation. $SE = s 1 + 1/ n = s 5 /2 = \frac{5}{2 10} = \frac{1}{2 2} = \frac{2}{4} \approx 0.354$ . Margin $4.303 \times 0.354 \approx 1.52$ ; interval $(1.48, 4.52)$ .
The difference is the extra $1$ under the root: $Var (Y_{0} - \overset{y}{^}_{0}) = σ^{2} + Var (\overset{y}{^}_{0})$ , the $σ^{2}$ being the new observation's own noise $ε_{0}$ , independent of everything used to fit. The ratio of half-widths is
$\frac{1 + 1/4}{1/4} = 5 \approx 2.24,$
and it would be even larger for bigger $n$ , since the numerator has a floor at $1$ while the denominator tends to $0$ .
(d) The $F$ check. $MSR = SSR /1 = 9.8$ and $MSE = SSE / (n - 2) = 0.2/2 = 0.1$ , so $F = 9.8/0.1 = 98$ . From (a), $t^{2} = (72)^{2} = 49 \times 2 = 98$ . $✓$ The two tests are the same test: the model comparison between the fitted line and the intercept-only model, with $F_{1, ν} = t_{ν}^{2}$ .
8.Critique
A student fits $\overset{y}{^} = - 0.5 + 0.7 x$ to the four points $(2, 1), (4, 2), (6, 4), (8, 5)$ , obtaining $R^{2} = 0.98$ , $s = 1/ 10 \approx 0.316$ , $S_{xx} = 20$ , $\overset{x}{ˉ} = 5$ , and writes the following report.
Step 1. Since $R^{2} = 0.98$ , the line explains $98%$ of the variation, so the relationship is certainly real and no test of the slope is required.
Step 2. A $95%$ interval for the value of $y$ that will be observed next time $x = 5$ is $3 \pm 4.303 \times 0.316 \times 1/4 = 3 \pm 0.68$ .
Step 3. The interval in Step 2 shrinks to zero width as $n \to \infty$ , so with enough data the next observation at $x = 5$ could be predicted exactly.
Step 4. Because the fitted line passes through $(\overset{x}{ˉ}, \overset{y}{ˉ})$ , the interval in Step 2 has the same width at every $x_{0}$ .
You may use $t_{0.025, 2} = 4.303$ , $2 \approx 1.414$ , $7 \approx 2.646$ and $10 \approx 3.162$ .
1. (a)Identify every flaw. For each, say which theorem, term or distinction is being misused. (One step is correct; say which.)
2. (b)Give the correct $95%$ interval for the next observation at $x_{0} = 5$ , and the correct $95%$ interval for the mean response at $x_{0} = 8$ .
Solution
(a) Four steps, three flaws.
Step 1 confuses $R^{2}$ with a test. $R^{2}$ is a descriptive ratio of sums of squares; it has no sampling distribution attached and says nothing about whether $β_{1} = 0$ is compatible with the data. The theorem Sampling distribution of the slope is what settles that, and it needs $n$ : on tiny data a large $R^{2}$ can coexist with an insignificant slope. As it happens the test here gives $t = 72 \approx 9.9$ against $4.303$ and the slope is significant — but that conclusion comes from the test, not from the $0.98$ . The number is right; the reasoning is not.
Step 2 uses the wrong interval. The formula $s 1/ n + (x_{0} - \overset{x}{ˉ})^{2} / S_{xx}$ is the standard error of $\overset{y}{^}_{0}$ as an estimate of the mean response $E (Y ∣ x_{0})$ . A single new observation has its own error $ε_{0}$ , and by the theorem Confidence and prediction intervals the variance of $Y_{0} - \overset{y}{^}_{0}$ is $σ^{2} (1 + 1/ n + (x_{0} - \overset{x}{ˉ})^{2} / S_{xx})$ — the $1$ is missing. The interval quoted is a confidence interval for the mean masquerading as a prediction interval, and it is far too narrow.
Step 3 is wrong for the interval it claims to be about. The mean-response interval does shrink to zero width, because $1/ n \to 0$ . The prediction interval does not: its half-width tends to $t s \to 1.96 σ$ , the floor set by the noise in the observation being predicted, which no amount of data about other observations can reduce. A future observation can never be predicted exactly by a model with $σ > 0$ .
Step 4 is correct in premise and wrong in conclusion. The line does pass through the point of means — but that is exactly why the width varies: the line pivots about $(\overset{x}{ˉ}, \overset{y}{ˉ})$ , so slope error has no effect at $\overset{x}{ˉ}$ and a growing effect away from it. The term $(x_{0} - \overset{x}{ˉ})^{2} / S_{xx}$ is zero at $x_{0} = 5$ and positive everywhere else; both bands are narrowest at $\overset{x}{ˉ}$ and flare outward.
(b) The correct intervals.
New observation at $x_{0} = 5$ . $\overset{y}{^}_{0} = 3$ and
$SE = s 1 + \frac{1}{4} + 0 = \frac{1}{10} \cdot \frac{5}{2} = \frac{1}{2 2} = \frac{2}{4} \approx 0.354,$
so the interval is $3 \pm 4.303 \times 0.354 = 3 \pm 1.52$ , that is $(1.48, 4.52)$ — more than twice as wide as the student's.
Mean response at $x_{0} = 8$ . $\overset{y}{^}_{0} = - 0.5 + 5.6 = 5.1$ , and $(x_{0} - \overset{x}{ˉ})^{2} / S_{xx} = 9/20$ , so
$SE = s \frac{1}{4} + \frac{9}{20} = \frac{1}{10} \frac{14}{20} = \frac{7}{100} = \frac{7}{10} \approx 0.265,$
giving $5.1 \pm 4.303 \times 0.265 = 5.1 \pm 1.14$ , that is $(3.96, 6.24)$ . Compare the half-width $0.68$ at the centre: moving three units from $\overset{x}{ˉ}$ has widened the mean-response interval by the factor $0.7/0.25 = 2.8 \approx 1.67$ , which is Step 4 refuted numerically.
9.Computation
Write the simple regression of $y$ on $x$ for the four points $(2, 1), (4, 2), (6, 4), (8, 5)$ in matrix form, with $X$ having a column of ones and a column of $x$ values, so that $p = 2$ .
1. (a)Write out $X^{⊤} X$ and $X^{⊤} y$ , invert the first, and solve the normal equations. Confirm the scalar answer $b_{0} = - 0.5$ , $b_{1} = 0.7$ , and check that the determinant equals $n S_{xx}$ .
2. (b)Read $Var (\hat{β}_{1})$ , $Var (\hat{β}_{0})$ and $Cov (\hat{β}_{0}, \hat{β}_{1})$ off $σ^{2} (X^{⊤} X)^{- 1}$ , and check the first two against $σ^{2} / S_{xx}$ and $σ^{2} (\frac{1}{n} + \overset{x}{ˉ}^{2} / S_{xx})$ .
3. (c)The $i$ -th diagonal entry of the hat matrix is $h_{ii} = x_{i}^{⊤} (X^{⊤} X)^{- 1} x_{i}$ where $x_{i}^{⊤} = (1, x_{i})$ is the $i$ -th row of $X$ . Compute all four $h_{ii}$ , verify that they sum to $2$ , and then prove that $tr (H) = p$ for every design matrix of full column rank. You may use $tr (AB) = tr (BA)$ .
Solution
(a) The normal equations by hand. With $\sum x_{i} = 20$ , $\sum x_{i}^{2} = 4 + 16 + 36 + 64 = 120$ , $\sum y_{i} = 12$ and $\sum x_{i} y_{i} = 2 + 8 + 24 + 40 = 74$ ,
$X^{⊤} X = (42020120), X^{⊤} y = (1274) .$
The determinant is $4 \times 120 - 2 0^{2} = 480 - 400 = 80$ , and $n S_{xx} = 4 \times 20 = 80$ . $✓$ (It is $n \sum x_{i}^{2} - (\sum x_{i})^{2} = n S_{xx}$ in general, the computational formula for the variance.) So
$(X^{⊤} X)^{- 1} = \frac{1}{80} (120 - 20 - 20 4),$
and by the theorem The normal equations and the least-squares solution,
$\hat{β} = \frac{1}{80} (120 \cdot 12 - 20 \cdot 74 - 20 \cdot 12 + 4 \cdot 74) = \frac{1}{80} (1440 - 1480 - 240 + 296) = \frac{1}{80} (- 40 56) = (- 0.5 0.7) . ✓$
(b) The covariance matrix. By the theorem Mean and variance of the coefficient estimates, $Var (\hat{β}) = σ^{2} (X^{⊤} X)^{- 1}$ , so
$Var (\hat{β}_{1}) = \frac{4}{80} σ^{2} = \frac{σ ^{2}}{20} = \frac{σ ^{2}}{S _{xx}}, ✓ Var (\hat{β}_{0}) = \frac{120}{80} σ^{2} = \frac{3}{2} σ^{2},$
and the check: $\frac{1}{n} + \overset{x}{ˉ}^{2} / S_{xx} = \frac{1}{4} + \frac{25}{20} = \frac{1}{4} + \frac{5}{4} = \frac{3}{2}$ . $✓$ (This is the mean-response variance at $x_{0} = 0$ , since $\hat{β}_{0} = \overset{y}{^} (0)$ .) The off-diagonal entry gives $Cov (\hat{β}_{0}, \hat{β}_{1}) = - \frac{20}{80} σ^{2} = - \frac{σ ^{2}}{4}$ , which is $- \overset{x}{ˉ} σ^{2} / S_{xx} = - 5 σ^{2} /20$ : with a positive $\overset{x}{ˉ}$ , overestimating the slope forces the intercept down, because the line must still pass through the point of means.
(c) Leverages and their sum. For row $(1, x)$ ,
$h = \frac{1}{80} (1 x) (120 - 20 - 20 4) (1 x) = \frac{120 - 40 x + 4 x ^{2}}{80} = \frac{x ^{2} - 10 x + 30}{20} = \frac{( x - 5 ) ^{2} + 5}{20} = \frac{1}{4} + \frac{( x - x ˉ ) ^{2}}{S _{xx}},$
which is the familiar bracket from the confidence-interval formula. Evaluating: at $x = 2$ and $x = 8$ , $h = \frac{1}{4} + \frac{9}{20} = 0.7$ ; at $x = 4$ and $x = 6$ , $h = \frac{1}{4} + \frac{1}{20} = 0.3$ . Sum: $0.7 + 0.3 + 0.3 + 0.7 = 2 = p$ . $✓$ The end points carry more leverage — the line is more strongly pulled by them.
Proof that $tr (H) = p$ . Using the cyclic property with $A = X$ and $B = (X^{⊤} X)^{- 1} X^{⊤}$ ,
$tr (H) = tr (X (X^{⊤} X)^{- 1} X^{⊤}) = tr ((X^{⊤} X)^{- 1} X^{⊤} X) = tr (I_{p}) = p . ■$
Geometrically, by the theorem Least squares is orthogonal projection, $H$ projects onto a $p$ -dimensional subspace, and the trace of a projection is the dimension of its range. The total leverage $p$ is fixed by the model; the data decide only how it is shared out.
10.Proof
Two predictors and the response have all been centred, so $\sum_{i} x_{i 1} = \sum_{i} x_{i 2} = \sum_{i} y_{i} = 0$ and the intercept may be dropped: the design matrix $X$ has just the two columns $x_{1}, x_{2}$ . Write $S_{11} = \sum_{i} x_{i 1}^{2}$ , $S_{22} = \sum_{i} x_{i 2}^{2}$ , $S_{12} = \sum_{i} x_{i 1} x_{i 2}$ (all with $S_{11}, S_{22} > 0$ ) and $r_{12} = S_{12} / S_{11} S_{22}$ . Assume $Var (ε) = σ^{2} I$ .
1. (a)Write out $X^{⊤} X$ and prove that it is invertible if and only if $∣ r_{12} ∣ < 1$ . Relate this to full column rank.
2. (b)Compute $(X^{⊤} X)^{- 1}$ and prove that $Var (\hat{β}_{1}) = \frac{σ ^{2}}{S _{11} ( 1 - r _{12}^{2} )}$ .
3. (c)Regress $x_{1}$ on $x_{2}$ alone (through the origin, since both are centred). Compute the residual sum of squares of that auxiliary fit and show $R_{1}^{2} = r_{12}^{2}$ . Conclude that $VIF_{1} = 1/ (1 - r_{12}^{2})$ and that (b) is exactly the variance-inflation formula in this case.
4. (d)If $r_{12} = 0.8$ , by what factor is the standard error of $\hat{β}_{1}$ larger than it would be with an uncorrelated second predictor, and how many times as many observations would compensate? Say what happens to $\hat{y}$ and to the bias of $\hat{β}_{1}$ as $r_{12} \to 1$ .
Solution
(a) The Gram matrix and its determinant.
$X^{⊤} X = (x_{1}^{⊤} x_{1} x_{2}^{⊤} x_{1} x_{1}^{⊤} x_{2} x_{2}^{⊤} x_{2}) = (S_{11} S_{12} S_{12} S_{22}),$
with determinant $S_{11} S_{22} - S_{12}^{2} = S_{11} S_{22} (1 - r_{12}^{2})$ , using $S_{12}^{2} = r_{12}^{2} S_{11} S_{22}$ . Since $S_{11} S_{22} > 0$ , the determinant is non-zero exactly when $r_{12}^{2} \neq = 1$ . By Cauchy–Schwarz, $S_{12}^{2} \leq S_{11} S_{22}$ , so $∣ r_{12} ∣ \leq 1$ always; hence invertible $⟺ ∣ r_{12} ∣ < 1$ . $■$
The equality case of Cauchy–Schwarz is proportionality, $x_{2} = c x_{1}$ , which is precisely failure of full column rank. So this is the theorem The normal equations and the least-squares solution in two dimensions: $∣ r_{12} ∣ = 1$ means the two columns carry the same information and there is no unique fit.
(b) Invert and read off the variance. For a $2 \times 2$ matrix,
$(X^{⊤} X)^{- 1} = \frac{1}{S _{11} S _{22} ( 1 - r _{12}^{2} )} (S_{22} - S_{12} - S_{12} S_{11}) .$
By the theorem Mean and variance of the coefficient estimates, $Var (\hat{β}_{1}) = σ^{2} [(X^{⊤} X)^{- 1}]_{11}$ , so
$Var (\hat{β}_{1}) = σ^{2} \cdot \frac{S _{22}}{S _{11} S _{22} ( 1 - r _{12}^{2} )} = \frac{σ ^{2}}{S _{11} ( 1 - r _{12}^{2} )} . ■$
With $r_{12} = 0$ this is $σ^{2} / S_{11}$ , the simple-regression variance; every non-zero $r_{12}$ inflates it.
(c) The auxiliary regression. Regressing $x_{1}$ on $x_{2}$ through the origin minimises $\sum_{i} (x_{i 1} - c x_{i 2})^{2} = S_{11} - 2 c S_{12} + c^{2} S_{22}$ , a quadratic in $c$ with minimum at $c = S_{12} / S_{22}$ . Substituting,
$SS E_{aux} = S_{11} - 2 \frac{S _{12}^{2}}{S _{22}} + \frac{S _{12}^{2}}{S _{22}} = S_{11} - \frac{S _{12}^{2}}{S _{22}} = S_{11} (1 - \frac{S _{12}^{2}}{S _{11} S _{22}}) = S_{11} (1 - r_{12}^{2}) .$
Because $x_{1}$ is centred its total sum of squares is $S_{11}$ , so
$R_{1}^{2} = 1 - \frac{SS E _{aux}}{S _{11}} = 1 - (1 - r_{12}^{2}) = r_{12}^{2},$
and $VIF_{1} = 1/ (1 - R_{1}^{2}) = 1/ (1 - r_{12}^{2})$ . Then (b) reads
$Var (\hat{β}_{1}) = \frac{σ ^{2}}{S _{11}} \cdot \frac{1}{1 - R _{1}^{2}} = \frac{σ ^{2}}{S _{11}} VIF_{1},$
which is the theorem Variance inflation with $p - 1 = 2$ predictors, proved rather than quoted. $■$ Note also that $SS E_{aux} = ∥ r_{1} ∥^{2}$ is the squared length of the part of $x_{1}$ not explained by $x_{2}$ , and $Var (\hat{β}_{1}) = σ^{2} /∥ r_{1} ∥^{2}$ : the coefficient is estimated from the leftover part alone.
(d) Numbers, and the limit. With $r_{12} = 0.8$ , $VIF_{1} = 1/ (1 - 0.64) = 1/0.36 = 25/9$ . Standard errors scale as $VIF$ , so $SE (\hat{β}_{1})$ is $5/3$ times what it would be with an uncorrelated predictor. Variance falls as $1/ n$ at fixed spread, so compensating would take $25/9 \approx 2.8$ times as many observations.
As $r_{12} \to 1$ the variance of $\hat{β}_{1}$ tends to infinity: the two coefficients become individually meaningless, typically large and of opposite sign. But two things do not break. $E (\hat{β}_{1}) = β_{1}$ throughout, by the same theorem — inflation is variance, never bias. And the fitted values $\hat{y} = Hy$ depend only on the column space of $X$ , which is the same plane for every $r_{12} < 1$ ; so prediction within the range of the data is unaffected. The data know what the pair predicts; they cannot say which predictor deserves the credit.
11.Critique
A model with an intercept and two predictors $x_{1}, x_{2}$ is fitted to $n = 33$ observations, giving $SST = 160$ and $SSE = 60$ . The two coefficients have $t$ statistics $1.2$ and $1.0$ , and each predictor has $R_{j}^{2} = 0.96$ when regressed on the other. Refitting with $x_{1}$ alone gives $SSE = 62$ and a $t$ statistic of $7.0$ for $x_{1}$ . Critical values: $t_{0.025, 30} = 2.042$ , $t_{0.025, 31} = 2.040$ , $F_{0.05, 2, 30} = 3.32$ .
The analyst reports:
Step 1. $R^{2} = 1 - 60/160 = 0.625$ , so the model accounts for $62.5%$ of the variation.
Step 2. Neither $∣ t ∣$ exceeds $2.042$ , so neither predictor is associated with $y$ ; the $62.5%$ must be an artefact.
Step 3. The variance inflation factors are $25$ , which means each coefficient estimate is biased by a factor of about $25$ and cannot be trusted in any case.
Step 4. Refitting with $x_{1}$ alone gives $t = 7.0$ , so the effect of $x_{1}$ is now established at $p < 0.001$ and $x_{2}$ was rightly discarded; since $SSE$ rose only from $60$ to $62$ , the one-predictor model also has the higher adjusted $R^{2}$ .
1. (a)Compute the overall $F$ statistic for the two-predictor model, the nested $F$ for dropping $x_{2}$ , and the adjusted $R^{2}$ of both models, all exactly.
2. (b)Identify every flaw in Steps 2–4, naming the theorem, remark or pitfall each one violates, and state what the report should have concluded instead.
Solution
(a) The numbers the analyst did not compute.
Overall $F$ . The reduced model is intercept-only, with $SS E_{red} = SST = 160$ and $q = 1$ ; the full model has $p = 3$ , so $n - p = 30$ . By the theorem Nested model comparison,
$F = \frac{( 160 - 60 ) / ( 3 - 1 )}{60/30} = \frac{50}{2} = 25,$
against $F_{0.05, 2, 30} = 3.32$ . Overwhelmingly significant: the pair of predictors explains far more than chance would.
Nested $F$ for dropping $x_{2}$ . $F = \frac{( 62 - 60 ) /1}{60/30} = \frac{2}{2} = 1$ , and indeed $F = t_{2}^{2} = 1. 0^{2}$ , as $F_{1, ν} = t_{ν}^{2}$ requires. $✓$ (Likewise for the refit, $F = \frac{( 160 - 62 ) /1}{62/31} = \frac{98}{2} = 49 = 7. 0^{2}$ . $✓$ )
Adjusted $R^{2}$ . With $SST / (n - 1) = 160/32 = 5$ :
$two predictors: 1 - \frac{60/30}{5} = 1 - \frac{2}{5} = 0.6; one predictor: 1 - \frac{62/31}{5} = 1 - \frac{2}{5} = 0.6.$
The two are equal — $s^{2} = 2$ in both models. This is the proposition Adjusted R-squared penalises useless predictors sitting exactly on its boundary: adding a predictor raises $R_{adj}^{2}$ if and only if $∣ t ∣ > 1$ , and $t_{2} = 1.0$ is the break-even case.
(b) The flaws.
Step 2: insignificant $t$ 's do not mean irrelevant predictors. With $R_{j}^{2} = 0.96$ the predictors are nearly interchangeable, and by the theorem Variance inflation each standard error is $25 = 5$ times what it would be for an uncorrelated predictor. The data cannot apportion credit between $x_{1}$ and $x_{2}$ , so each coefficient, holding the other fixed, is poorly determined. The joint test in (a) says the pair carries a great deal of information, $F = 25$ . This is the puzzle in the remark What multicollinearity does and does not break: a highly significant $F$ with no significant coefficient is the signature of collinearity, not of an artefact. The $62.5%$ is real.
Step 3: inflation is variance, not bias. The theorem Mean and variance of the coefficient estimates gives $E (\hat{β}) = β$ whatever the correlation among predictors; Gauss–Markov unbiasedness is untouched. What a VIF of $25$ says is $Var (\hat{β}_{j}) = 25 σ^{2} / S_{jj}$ — the estimate is right on average and imprecise, which is the opposite of “cannot be trusted”. Nor is the factor $25$ applied to the estimate: it multiplies the variance, and the standard error by $5$ .
Step 4 contains three errors. First, the $p$ -value from the refit is not valid: the model was chosen by looking at these same data, and a coefficient that survived a selection step is biased away from zero because it survived — the pitfall on stepwise selection. Since $x_{1}$ and $x_{2}$ are near-duplicates, dropping $x_{1}$ instead would have produced an equally impressive $t$ for $x_{2}$ ; the choice was arbitrary and the “ $p < 0.001$ ” describes the selection, not $x_{1}$ . Second, the adjusted $R^{2}$ claim is simply false — (a) shows the two values are equal, so the criterion is indifferent between the models. Third, “the effect of $x_{1}$ ” is causal language for what is a partial association; nothing in the fit licenses it (the pitfall on coefficients and causes).
What should have been concluded. The two predictors jointly predict $y$ well ( $F = 25$ , $R^{2} = 0.625$ ) and the data cannot say which of them matters. If the goal is prediction inside the observed range, keep both — the fitted values are unaffected by collinearity. If the goal is interpretation, either combine the two into one index, or collect data in which they vary separately, and report any selection that was done as exploratory, with standard errors assessed on data not used to select.
12.Computation
For the four points $(2, 1), (4, 2), (6, 4), (8, 5)$ : $\overset{x}{ˉ} = 5$ , $\overset{y}{ˉ} = 3$ , $S_{xx} = 20$ , $S_{yy} = 10$ , $S_{x y} = 14$ , $r^{2} = 0.98$ , $s^{2} = 0.1$ , and the regression of $y$ on $x$ is $\overset{y}{^} = - 0.5 + 0.7 x$ . You may use $46 \approx 6.78$ and $115 \approx 10.72$ .
1. (a)Fit the regression of $x$ on $y$ . Show that it is not the algebraic inverse of $\overset{y}{^} = - 0.5 + 0.7 x$ , and prove in general that the product of the two slopes is $r^{2}$ . When do the two lines coincide?
2. (b)Compute the standard error of the estimated mean response at $x_{0} = 5$ and at $x_{0} = 20$ , exactly, and their ratio. Then explain why even the wider interval at $x_{0} = 20$ understates the real uncertainty there.
3. (c)Someone reads the slope $0.7$ as “increasing $x$ by one unit will increase $y$ by $0.7$ ”. Name three distinct mechanisms that could produce this slope without that being true, and say which of them a residual plot can detect.
Solution
(a) The other regression. Minimising horizontal rather than vertical squared distances, the roles of $x$ and $y$ swap:
$slope = \frac{S _{x y}}{S _{yy}} = \frac{14}{10} = \frac{7}{5}, intercept = \overset{x}{ˉ} - \frac{7}{5} \overset{y}{ˉ} = 5 - \frac{21}{5} = \frac{4}{5},$
so $\overset{x}{^} = 0.8 + 1.4 y$ . Solving $\overset{y}{^} = - 0.5 + 0.7 x$ for $x$ instead gives $x = (y + 0.5) /0.7 = \frac{5}{7} + \frac{10}{7} y$ , with slope $10/7 \approx 1.43 \neq = 1.4$ . The two are different lines; each is optimal for its own direction of error and neither can be obtained from the other.
Proof of the product. $b_{y ∣ x} b_{x ∣ y} = \frac{S _{x y}}{S _{xx}} \cdot \frac{S _{x y}}{S _{yy}} = \frac{S _{x y}^{2}}{S _{xx} S _{yy}} = r^{2}$ . $■$ Here $0.7 \times 1.4 = 0.98 = r^{2}$ . $✓$ The inverse of the first line has slope $1/ b_{y ∣ x}$ , and the second line has slope $r^{2} / b_{y ∣ x}$ ; they agree exactly when $r^{2} = 1$ , that is when the points are collinear (the theorem Correlation lies between $- 1$ and $1$ ). Otherwise the $x$ -on- $y$ line is always the flatter of the two when drawn in the $(x, y)$ plane — regression to the mean in both directions at once.
(b) The cost of leaving the data. By the theorem Confidence and prediction intervals, $SE (\overset{y}{^}_{0}) = s \frac{1}{n} + (x_{0} - \overset{x}{ˉ})^{2} / S_{xx}$ with $s = 1/ 10$ .
At $x_{0} = 5 = \overset{x}{ˉ}$ : $SE = s 1/4 = \frac{1}{2 10} = \frac{10}{20} \approx 0.16$ .
At $x_{0} = 20$ : $(x_{0} - \overset{x}{ˉ})^{2} / S_{xx} = 225/20 = 45/4$ , so $\frac{1}{4} + \frac{45}{4} = \frac{23}{2}$ and
$SE = \frac{1}{10} \frac{23}{2} = \frac{23}{20} = \frac{115}{10} \approx 1.07.$
Ratio: $\frac{23/2}{1/4} = 46 \approx 6.8$ . Twelve units beyond the last observation, the interval for the mean is nearly seven times as wide as at the centre, because the fitted line pivots about $(\overset{x}{ˉ}, \overset{y}{ˉ})$ and slope error is amplified by the lever arm $x_{0} - \overset{x}{ˉ}$ .
Even so it understates the uncertainty, because the formula is derived assuming the model holds at $x_{0} = 20$ : that $E (Y ∣ x)$ is still $β_{0} + β_{1} x$ out there. The four observations, all in $[2, 8]$ , contain no information about that. A relationship linear over $[2, 8]$ may bend, saturate or reverse beyond it — the parabola of an earlier question fits a line with $R^{2} \approx 0.92$ and then departs from it without limit — and no interval built from $s$ and $S_{xx}$ can price a failure of the model's form. The widening is the part of the risk the mathematics can see; the rest is invisible to it.
(c) Why a coefficient is not an effect. The slope $0.7$ describes the conditional mean of $y$ given $x$ in these data. Three mechanisms produce such a slope with no causal path from $x$ to $y$ :
Confounding — a third variable drives both $x$ and $y$ , so they move together without either moving the other.
Reverse causation — $y$ drives $x$ . The algebra is symmetric: (a) shows the regression of $x$ on $y$ describes the same four points equally well, and nothing in the fit says which way any influence runs.
Selection — the four points were included because of their values of both variables, creating an association among those observed that need not exist in the population.
A residual plot can detect none of them. It checks the LINE conditions — curvature, changing spread, gross non-normality — which concern how well a line describes the conditional mean. All three mechanisms are perfectly compatible with a clean linear conditional mean and an immaculate residual plot. What licenses a causal reading is the design of the study, not the diagnostics of the fit.

← Back to the question sheet