Print this page and work through it on paper. Revealed answers are never printed, so you can check one question and still print a clean sheet.

1. 1.Computation · 6 min

   A linear probability model is fitted by least squares to whether a household owns a car ($Y=1$) against household income $x$ in thousands, giving

   $$\hat{p}=-\frac{1}{5}+\frac{1}{20}x.$$

   1. (a)Find the range of incomes over which the fitted value is a valid probability, and evaluate the fit at $x=2$ and at $x=30$.
   2. (b)For a household with $x=8$, write down the two values the error $\epsilon =Y-p$ can take and the probability of each, and compute the error variance. Repeat for $x=14$.
   3. (c)Say which of the three failures of the linear probability model each of (a) and (b) illustrates. Which of the three cannot be repaired by adjusting standard errors, and why not?
   Show the solution to question 1

   (a) Solve the two inequalities. $\hat{p}\ge 0$ requires $\frac{x}{20}\ge \frac{1}{5}$, that is $x\ge 4$; and $\hat{p}\le 1$ requires $\frac{x}{20}\le \frac{6}{5}$, that is $x\le 24$. The fit is a probability only for incomes between $\$4,000$ and $\$24,000$.

   At $x=2$: $\hat{p}=-\frac{1}{5}+\frac{1}{10}=-\frac{1}{10}$, a negative probability. At $x=30$: $\hat{p}=-\frac{1}{5}+\frac{3}{2}=\frac{13}{10}$, a probability of $130\%$. Both are statements that cannot be true of any household.

   (b) The error is two-valued. At $x=8$ the model probability is $p=-\frac{1}{5}+\frac{8}{20}=\frac{1}{5}$. Since $Y$ is Bernoulli$(p)$, the error $\epsilon =Y-p$ equals $1-p=\frac{4}{5}$ with probability $p=\frac{1}{5}$, and equals $-p=-\frac{1}{5}$ with probability $1-p=\frac{4}{5}$. Its variance is

   $$p(1-p)=\frac{1}{5}\cdot \frac{4}{5}=\frac{4}{25}.$$

   Check directly: $E(\epsilon )=\frac{4}{5}\cdot \frac{1}{5}-\frac{1}{5}\cdot \frac{4}{5}=0$ and $E({\epsilon }^2)=\frac{16}{25}\cdot \frac{1}{5}+\frac{1}{25}\cdot \frac{4}{5}=\frac{20}{125}=\frac{4}{25}$. ✓

   At $x=14$: $p=-\frac{1}{5}+\frac{14}{20}=\frac{1}{2}$. The error is $\pm \frac{1}{2}$, each with probability $\frac{1}{2}$, and the variance is $\frac{1}{2}\cdot \frac{1}{2}=\frac{1}{4}$.

   (c) Rank the failures. Part (a) is the range failure: the fitted line leaves $[0,1]$. Part (b) shows the other two at once. The variance is $\frac{4}{25}$ at one income and $\frac{1}{4}$ at another, so the errors are heteroscedastic; and at each income the error takes only two values, so it is not normal.

   Heteroscedasticity and non-normality are the familiar complaints: by Gauss–Markov the least-squares estimates stay unbiased, only the standard errors are wrong, and weighted least squares or robust standard errors repair them. The range failure is different in kind. A fitted probability of $\frac{13}{10}$ is not an imprecise estimate of something — the quantity being estimated cannot exceed $1$ — so no confidence interval around it helps. That failure forces a new functional form, which is why the log-odds is modelled instead of the probability.

2. 2.Proof · 7 min

   Let $\sigma (z)=\frac{1}{1+e^{-z}}$. Prove each statement directly from this formula.

   1. (a)$\sigma (-z)=1-\sigma (z)$ for every real $z$, and consequently ${\sigma }^{\prime }(z)=\sigma (z)\sigma (-z)$.
   2. (b)${\sigma }^{\prime }(z)\le \frac{1}{4}$ for every $z$, with equality only at $z=0$.
   3. (c)${\sigma }^{\prime \prime }(z)={\sigma }^{\prime }(z)(1-2\sigma (z))$. Deduce where $\sigma$ is convex, where it is concave, and where its inflection point lies, and say what this means for the effect of a predictor on a probability.
   Show the solution to question 2

   (a) Symmetry, then the derivative. By the formula, $\sigma (-z)=\frac{1}{1+e^z}$. Multiply numerator and denominator by $e^{-z}$:

   $$\sigma (-z)=\frac{e^{-z}}{e^{-z}+1}.$$

   On the other side, $1-\sigma (z)=\frac{(1+e^{-z})-1}{1+e^{-z}}=\frac{e^{-z}}{1+e^{-z}}$, the same expression. So $\sigma (-z)=1-\sigma (z)$.

   For the derivative, write $\sigma (z)=(1+e^{-z}{)}^{-1}$ and apply the chain rule:

   $${\sigma }^{\prime }(z)=-(1+e^{-z}{)}^{-2}\cdot (-e^{-z})=\frac{1}{1+e^{-z}}\cdot \frac{e^{-z}}{1+e^{-z}}=\sigma (z)\sigma (-z),$$

   using the expression for $\sigma (-z)$ just found. Equivalently ${\sigma }^{\prime }=\sigma (1-\sigma )$.

   (b) Complete the square. Put $p=\sigma (z)$. Since $e^{-z}>0$, the denominator exceeds $1$, so $0<p<1$. Then

   $${\sigma }^{\prime }(z)=p(1-p)=\frac{1}{4}-{\left(p-\frac{1}{2}\right)}^2\le \frac{1}{4},$$

   with equality exactly when $p=\frac{1}{2}$. Now $\sigma (z)=\frac{1}{2}$ means $1+e^{-z}=2$, so $e^{-z}=1$ and $z=0$. Both factors of ${\sigma }^{\prime }$ are positive, so $\sigma$ is strictly increasing and takes the value $\frac{1}{2}$ only once, at $z=0$. Hence the bound $\frac{1}{4}$ is attained only there.

   (c) Differentiate ${\sigma }^{\prime }=\sigma -{\sigma }^2$. By the product/chain rule,

   $${\sigma }^{\prime \prime }={\sigma }^{\prime }-2\sigma {\sigma }^{\prime }={\sigma }^{\prime }(1-2\sigma ).$$

   Since ${\sigma }^{\prime }>0$ everywhere, the sign of ${\sigma }^{\prime \prime }$ is the sign of $1-2\sigma (z)$. For $z<0$ we have $\sigma (z)<\frac{1}{2}$ (strictly increasing, $\sigma (0)=\frac{1}{2}$), so ${\sigma }^{\prime \prime }>0$ and $\sigma$ is convex there; for $z>0$, ${\sigma }^{\prime \prime }<0$ and $\sigma$ is concave. The sign changes at $z=0$, so the inflection point is $\left(0,\frac{1}{2}\right)$ — the same point at which, by (b), the slope is largest.

   For a logistic model this says: a unit change in a predictor moves the probability most when the current probability is near $\frac{1}{2}$, and the effect shrinks towards zero as the probability approaches $0$ or $1$. The steepest possible probability change per unit of log-odds is $\frac{1}{4}$, and it is only available at the centre. $■$

3. 3.Computation · 7 min

   A logistic model for whether a seedling survives its first winter, in terms of its stem diameter $x$ in millimetres, is fitted as

   $$\mathrm{logit}(\hat{p})=-2\ln 3+(\ln 3)x.$$

   Give every answer exactly.

   1. (a)Compute the fitted odds and the fitted probability of survival at $x=0,1,2,3,4$.
   2. (b)State the odds ratio for a $1$ mm increase in diameter and for a $2$ mm increase, and explain why neither depends on the starting diameter.
   3. (c)Compute the change in fitted probability from $x=1$ to $2$, from $2$ to $3$, and from $3$ to $4$, and reconcile the pattern you find with (b).
   Show the solution to question 3

   (a) Work on the odds scale first. The linear predictor is $\eta =(x-2)\ln 3$, so the odds are

   $$\frac{\hat{p}}{1-\hat{p}}=e^{\eta }=e^{(x-2)\ln 3}=3^{x-2}.$$

   At $x=0,1,2,3,4$ this gives odds $\frac{1}{9},\frac{1}{3},1,3,9$. Converting with $\hat{p}=\frac{\text{odds}}{1+\text{odds}}$:

   $$\hat{p}=\frac{1}{10},\frac{1}{4},\frac{1}{2},\frac{3}{4},\frac{9}{10}.$$

   (For instance at $x=1$: $\frac{1/3}{1+1/3}=\frac{1/3}{4/3}=\frac{1}{4}$.)

   (b) Coefficients are log odds ratios. A $1$ mm increase adds ${\beta }_1=\ln 3$ to the log-odds, so it multiplies the odds by $e^{\ln 3}=3$; a $2$ mm increase adds $2\ln 3$ and multiplies them by $e^{2\ln 3}=9$. The reason it is independent of the starting point is that the odds are $e^{\eta }$ and $e^{\eta +{\beta }_1}=e^{\eta }{e}^{{\beta }_1}$: the ratio of new odds to old is $e^{{\beta }_1}$ whatever $\eta$ was. The table in (a) shows it — $\frac{1}{9}\to \frac{1}{3}\to 1\to 3\to 9$ is a factor of $3$ at every step.

   (c) The probability steps are not constant.

   $$\frac{1}{2}-\frac{1}{4}=\frac{1}{4},\frac{3}{4}-\frac{1}{2}=\frac{1}{4},\frac{9}{10}-\frac{3}{4}=\frac{3}{20}.$$

   The first two steps are equal because they are mirror images about $p=\frac{1}{2}$: $\sigma (-z)=1-\sigma (z)$ places $\frac{1}{4}$ and $\frac{3}{4}$ symmetrically. The third step is smaller because the curve is flattening as the probability nears $1$ — the same $\ln 3$ on the log-odds scale buys less and less probability. There is no contradiction with (b): the odds ratio is exactly $3$ at each step, and it is the probability that has no fixed exchange rate against the coefficient.

4. 4.Computation · 6 min

   Write $p_1$ for the probability of an outcome in a treated group and $p_2$ for the probability in a control group.

   1. (a)In a trial, $\frac{1}{4}$ of control patients recover and $\frac{1}{2}$ of treated patients recover. Compute the relative risk and the odds ratio for treatment. Which of the two is the logistic regression coefficient for treatment built from, and what is that coefficient exactly?
   2. (b)Repeat for a rare outcome: a side effect occurs in $\frac{1}{100}$ of controls and $\frac{2}{100}$ of treated patients.
   3. (c)Prove that in general $\text{OR}=\text{RR}\cdot \frac{1-p_2}{1-p_1}$, and deduce that whenever treatment raises the probability, the odds ratio exceeds the relative risk, with the gap small exactly when both probabilities are small. Confirm the formula against (a) and (b).
   Show the solution to question 4

   (a) The relative risk is $\text{RR}=\frac{p_1}{p_2}=\frac{1/2}{1/4}=2$. The odds are $\frac{1/2}{1/2}=1$ for treated and $\frac{1/4}{3/4}=\frac{1}{3}$ for control, so $\text{OR}=\frac{1}{1/3}=3$. Treatment doubles the risk but triples the odds.

   A logistic coefficient is a log odds ratio, never a log relative risk: with treatment coded $x=1$, ${\beta }_1=\ln \text{OR}=\ln 3$.

   (b) $\text{RR}=\frac{2/100}{1/100}=2$ again. The odds are $\frac{2/100}{98/100}=\frac{2}{98}=\frac{1}{49}$ and $\frac{1/100}{99/100}=\frac{1}{99}$, so

   $$\text{OR}=\frac{1/49}{1/99}=\frac{99}{49}\approx \mathrm{2.02.}$$

   For a rare outcome the two measures nearly agree; for the common outcome in (a) they did not.

   (c) Rearrange the definition.

   $$\text{OR}=\frac{p_1/(1-p_1)}{p_2/(1-p_2)}=\frac{p_1}{p_2}\cdot \frac{1-p_2}{1-p_1}=\text{RR}\cdot \frac{1-p_2}{1-p_1}.$$

   If $p_1>p_2$ then $1-p_2>1-p_1>0$, so the correction factor exceeds $1$ and $\text{OR}>\text{RR}$: an odds ratio reported in the language of risk always overstates a beneficial-looking effect. The factor $\frac{1-p_2}{1-p_1}$ tends to $1$ as $p_1,p_2\to 0$, which is the rare-outcome case in which the two agree.

   Checks: in (a) the factor is $\frac{3/4}{1/2}=\frac{3}{2}$ and $2\cdot \frac{3}{2}=3$ ✓; in (b) it is $\frac{99/100}{98/100}=\frac{99}{98}$ and $2\cdot \frac{99}{98}=\frac{99}{49}$ ✓.

5. 5.Critique · 8 min

   A logistic regression of loan default on years of employment $x$ reports ${\hat{\beta }}_1=-\frac{1}{2}$ with standard error $\frac{1}{5}$. The analyst's report contains the three claims below. Each is wrong. For each, name the error precisely, then give the correct statement with exact numbers. Use $2$ as the two-sided $5\%$ critical value wherever one is needed.

   1. (a)“Each extra year of employment lowers the probability of default by $0.5$, so five years of employment reduce it by $2.5$ — or to zero, whichever comes first.”
   2. (b)“The $95\%$ interval for ${\beta }_1$ is $-\frac{1}{2}\pm 2\cdot \frac{1}{5}=\left(-\frac{9}{10},-\frac{1}{10}\right)$. Exponentiating the point estimate and the half-width gives the odds-ratio interval $e^{-1/2}\pm e^{2/5}$.”
   3. (c)“The odds ratio per year is $e^{-1/2}\approx 0.61$, so a borrower with one more year of employment is about $39\%$ less likely to default.”
   Show the solution to question 5

   (a) A coefficient is a change in log-odds, not in probability. The correct multiplicative statement is: each extra year multiplies the odds of default by $e^{-1/2}$, and five years multiply them by $e^{-5/2}$. The probability change has no fixed value — it depends on where you start — and it can be bounded. By the mean value theorem, for some $\xi$ between $\eta$ and $\eta -\frac{1}{2}$,

   $$\sigma (\eta )-\sigma \left(\eta -\frac{1}{2}\right)=\frac{1}{2}{\sigma }^{\prime }(\xi )\le \frac{1}{2}\cdot \frac{1}{4}=\frac{1}{8},$$

   since ${\sigma }^{\prime }=\sigma (1-\sigma )\le \frac{1}{4}$. So no single year can move the probability by more than $\frac{1}{8}$, let alone $\frac{1}{2}$. For instance from $p=\frac{1}{2}$ (where $\eta =0$) one more year gives $p=\sigma (-\frac{1}{2})=\frac{1}{1+e^{1/2}}$, which exceeds $\frac{1}{3}$ because $e^{1/2}<2$: a drop of less than $\frac{1}{6}$. And the probability can never reach zero, since $\sigma$ takes values in $(0,1)$; it only approaches it.

   (b) The interval is built correctly and transformed incorrectly. On the log-odds scale ${\hat{\beta }}_1$ is approximately normal, so $\left(-\frac{9}{10},-\frac{1}{10}\right)$ is right. To move to the odds-ratio scale, apply $\exp$ to the two endpoints: because $\exp$ is strictly increasing, ${\beta }_1$ lies in the interval exactly when $e^{{\beta }_1}$ lies in $\left(e^{-9/10},e^{-1/10}\right)$, so the coverage is unchanged.

   The proposed interval $e^{-1/2}\pm e^{2/5}$ is not even a set of odds ratios: its lower end is $e^{-1/2}-e^{2/5}<0$, since $e^{2/5}>1>e^{-1/2}$, and an odds ratio is positive. The correct interval is not symmetric about $e^{-1/2}$; rather $e^{-9/10}\cdot e^{-1/10}=e^{-1}={\left(e^{-1/2}\right)}^2$, so $e^{-1/2}$ is its geometric centre. Consistency check: the Wald statistic is $z=\frac{-1/2}{1/5}=-\frac{5}{2}$, with $|z|>2$; the log-odds interval excludes $0$ and the odds-ratio interval excludes $1$, as they must.

   (c) An odds ratio is not a relative risk. “Less likely” is a statement about probabilities, and by the identity $\text{OR}=\text{RR}\cdot \frac{1-p_2}{1-p_1}$ the two agree only when default is rare. The correct sentence is “one more year multiplies the odds of default by $e^{-1/2}$”. To see the size of the discrepancy, take a borrower whose current probability is $\frac{1}{2}$: after one more year the odds are $e^{-1/2}$, so the probability is $\frac{1}{1+e^{1/2}}$ and the relative risk is

   $$\text{RR}=\frac{2}{1+e^{1/2}}>e^{-1/2},$$

   because $2e^{1/2}>1+e^{1/2}$ reduces to $e^{1/2}>1$. The risk falls by less than $39\%$; the odds fall by $39\%$. Only for a borrower whose default probability is already tiny would the two readings coincide.

6. 6.Computation · 10 min

   An intercept-only logistic model, $\mathrm{logit}(p_i)=\beta$ for every $i$, is fitted to $n=12$ binary observations of which $s=9$ are successes. You may use $e^{4/3}\approx 3.79$.

   1. (a)Write the log-likelihood $\ell (\beta )$ in terms of $s$, $n$ and $\beta$, show that the score equation reduces to $\sigma (\beta )=s/n$, and solve it exactly.
   2. (b)Starting from ${\beta }^{(0)}=0$, take one Newton step. Does it overshoot or undershoot the exact maximiser? Justify without a calculator.
   3. (c)Evaluate the Hessian at the maximiser and deduce the standard error of $\hat{\beta }$. Write the Wald statistic for $H_0:\beta =0$ exactly and decide the test with $2$ as the critical value.
   4. (d)Compute the deviance of the fitted model exactly.
   Show the solution to question 6

   (a) Sum the Bernoulli contributions. With every linear predictor equal to $\beta$,

   $$\ell (\beta )=\sum _{i=1}^n[y_i\beta -\ln (1+e^{\beta }\left)\right]=s\beta -n\ln (1+e^{\beta }).$$

   Differentiating, ${\ell }^{\prime }(\beta )=s-n\frac{e^{\beta }}{1+e^{\beta }}=s-n\sigma (\beta )$, which is the score equation ${\sum }_i(y_i-p_i)=0$ with all $p_i$ equal. Setting it to zero gives $\sigma (\hat{\beta })=\frac{s}{n}=\frac{3}{4}$, so $\hat{\beta }=\mathrm{logit}\left(\frac{3}{4}\right)=\ln \frac{3/4}{1/4}=\ln 3$. Because $\ell$ is strictly concave (its second derivative $-n\sigma (1-\sigma )$ is negative), this stationary point is the unique global maximum.

   (b) One Newton step. At ${\beta }^{(0)}=0$ every $p_i=\sigma (0)=\frac{1}{2}$. The gradient is $s-n\cdot \frac{1}{2}=9-6=3$ and the Hessian is $-n\sigma (1-\sigma )=-12\cdot \frac{1}{4}=-3$. Newton's update is

   $${\beta }^{(1)}={\beta }^{(0)}-\frac{{\ell }^{\prime }(0)}{{\ell }^{\prime \prime }(0)}=0-\frac{3}{-3}=1.$$

   The exact maximiser is $\ln 3$, and $\ln 3>1$ because $3>e$. So the step undershoots, landing at $1$ against the target $\ln 3\approx 1.099$ — most of the way in a single step, as expected near the optimum of a concave function; a second step would close almost all of the remaining gap.

   (c) Curvature at the maximiser. With $\sigma (\hat{\beta })=\frac{3}{4}$,

   $${\ell }^{\prime \prime }(\hat{\beta })=-n\cdot \frac{3}{4}\cdot \frac{1}{4}=-\frac{9}{4}.$$

   The observed information is $\frac{9}{4}$, and the standard error is its inverse square root: $SE(\hat{\beta })=\sqrt{\frac{4}{9}}=\frac{2}{3}$. (In general $SE=1/\sqrt{n\hat{p}(1-\hat{p})}$, the familiar binomial standard error transported to the log-odds scale.)

   The Wald statistic is $z=\frac{\hat{\beta }}{SE}=\frac{\ln 3}{2/3}=\frac{3}{2}\ln 3$. It exceeds $2$ exactly when $\ln 3>\frac{4}{3}$, that is when $3>e^{4/3}\approx 3.79$, which is false. So $z\approx 1.65<2$ and we do not reject $\beta =0$ at the $5\%$ level: nine successes in twelve trials is consistent with a fair coin.

   (d) Deviance. At $\hat{\beta }=\ln 3$ we have $1+e^{\hat{\beta }}=4$, so

   $$\ell (\hat{\beta })=9\ln 3-12\ln 4=9\ln 3-24\ln 2,D=-2\ell (\hat{\beta })=48\ln 2-18\ln 3.$$

   Check by the other route: for an intercept-only fit $\ell (\hat{\beta })=s\ln \frac{s}{n}+(n-s)\ln \frac{n-s}{n}=9\ln \frac{3}{4}+3\ln \frac{1}{4}=9\ln 3-18\ln 2-6\ln 2=9\ln 3-24\ln 2$ ✓. Numerically $D\approx 33.27-19.78=13.5$.

7. 7.Proof · 10 min

   A logistic model contains an intercept and a single binary predictor $x_i\in \{0,1\}$: $\mathrm{logit}(p_i)={\beta }_0+{\beta }_1{x}_i$. Write $n_1$ for the number of observations with $x_i=1$ and $s_1$ for the number of successes among them; define $n_0,s_0$ likewise for $x_i=0$. Assume $0<s_0<n_0$ and $0<s_1<n_1$.

   1. (a)Write out the two score equations ${\mathbf{X}}^{\top }(\mathbf{y}-\hat{\mathbf{p}})=0$ for this design, and deduce that at the maximum likelihood fit ${\sum }_{i:x_i=1}{\hat{p}}_i=s_1$ and ${\sum }_{i:x_i=0}{\hat{p}}_i=s_0$.
   2. (b)Hence prove that the fitted probability in each group equals that group's observed success proportion, and that ${\hat{\beta }}_1$ is exactly the logarithm of the sample odds ratio $\frac{s_1/(n_1-s_1)}{s_0/(n_0-s_0)}$.
   3. (c)In a study of $100$ patients, $40$ received a new drug and $30$ of them recovered; of the $60$ controls, $15$ recovered. Write down ${\hat{\beta }}_0$, ${\hat{\beta }}_1$ and the two fitted probabilities exactly. Then explain where the argument in (b) breaks if $s_1=n_1$.
   Show the solution to question 7

   (a) Two columns, two equations. The design matrix has a column of ones and the indicator column $(x_i)$. The score equations ${\mathbf{X}}^{\top }(\mathbf{y}-\hat{\mathbf{p}})=0$ are therefore

   $$\sum _{i=1}^n(y_i-{\hat{p}}_i)=0\text{and}\sum _{i=1}^n{x}_i(y_i-{\hat{p}}_i)=\sum _{i:x_i=1}(y_i-{\hat{p}}_i)=0.$$

   The second says ${\sum }_{x_i=1}{\hat{p}}_i={\sum }_{x_i=1}{y}_i=s_1$. Subtracting it from the first leaves ${\sum }_{x_i=0}(y_i-{\hat{p}}_i)=0$, so ${\sum }_{x_i=0}{\hat{p}}_i=s_0$. In words: with an intercept in the model, the fitted probabilities sum to the observed successes overall, and with an indicator in the model they do so within the indicated group as well.

   (b) Same predictor, same fitted probability. Every observation with $x_i=1$ has the same linear predictor ${\hat{\beta }}_0+{\hat{\beta }}_1$, hence the same fitted probability, call it ${\hat{p}}_{(1)}$; every observation with $x_i=0$ has ${\hat{p}}_{(0)}=\sigma ({\hat{\beta }}_0)$. The sums in (a) become $n_1{\hat{p}}_{(1)}=s_1$ and $n_0{\hat{p}}_{(0)}=s_0$, so

   $${\hat{p}}_{(1)}=\frac{s_1}{n_1},{\hat{p}}_{(0)}=\frac{s_0}{n_0}.$$

   Applying the logit, ${\hat{\beta }}_0=\mathrm{logit}\left(\frac{s_0}{n_0}\right)=\ln \frac{s_0}{n_0-s_0}$ and ${\hat{\beta }}_0+{\hat{\beta }}_1=\ln \frac{s_1}{n_1-s_1}$. Subtracting,

   $${\hat{\beta }}_1=\ln \frac{s_1}{n_1-s_1}-\ln \frac{s_0}{n_0-s_0}=\ln \frac{s_1/(n_1-s_1)}{s_0/(n_0-s_0)},$$

   the log of the sample odds ratio. This stationary point is the maximum: both groups are present, so $\mathbf{X}$ has full rank, and the assumption $0<s_j<n_j$ places every fitted probability strictly inside $(0,1)$, so the log-likelihood is strictly concave and its unique stationary point is the global maximiser. $■$

   (c) Numbers. ${\hat{p}}_{(1)}=\frac{30}{40}=\frac{3}{4}$ with odds $3$; ${\hat{p}}_{(0)}=\frac{15}{60}=\frac{1}{4}$ with odds $\frac{1}{3}$. Hence

   $${\hat{\beta }}_0=\ln \frac{1}{3}=-\ln 3,{\hat{\beta }}_1=\ln 3-(-\ln 3)=2\ln 3=\ln 9,$$

   an odds ratio of $9$ for the drug. Check: ${\hat{\beta }}_0+{\hat{\beta }}_1=\ln 3=\mathrm{logit}(\frac{3}{4})$ ✓.

   If $s_1=n_1$ — every treated patient recovers — then (a) demands ${\hat{p}}_{(1)}=1$, which no finite ${\hat{\beta }}_0+{\hat{\beta }}_1$ can deliver because $\sigma$ never reaches $1$. The score equations then have no solution: this is separation on the indicator, the likelihood increases without bound as ${\beta }_1\to \infty$, and the maximum likelihood estimate does not exist. The sample odds ratio $\frac{n_1/0}{\cdots }$ is infinite for the same reason.

8. 8.Computation · 8 min

   Three nested models of the same binary response are fitted. Model A has only an intercept; Model B adds one predictor; Model C adds three further predictors to B. Their deviances are $D_A=200$, $D_B=190$ and $D_C=184$. Critical values at the $5\%$ level: ${\chi }_1^2:3.841$, ${\chi }_3^2:7.815$, ${\chi }_4^2:9.488$.

   1. (a)Test B against A, C against B, and C against A. For each, state the statistic, its degrees of freedom, the theorem that justifies the reference distribution, and the conclusion.
   2. (b)Explain why $D_C\le D_B\le D_A$ had to hold whatever the data, and why the three statistics in (a) satisfy the identity you notice.
   3. (c)In Model B the predictor's coefficient is ${\hat{\beta }}_1=\frac{3}{5}$ with standard error $\frac{1}{5}$. Compute the Wald statistic and compare its square with the likelihood-ratio statistic for the same hypothesis. Should the two agree exactly?
   4. (d)Using $2$ as the critical value, give a $95\%$ confidence interval for the odds ratio per unit of the predictor, and show that it extends further above $e^{3/5}$ than below it, by an exact factor.
   Show the solution to question 8

   (a) Likelihood-ratio tests. For nested models differing by $k$ parameters, $D_{\text{red}}-D_{\text{full}}\to {\chi }_k^2$ under the null that the extra coefficients are zero — Wilks' theorem, quoted for twice the log-likelihood ratio.

   B against A: $200-190=10$ on $1$ degree of freedom; $10>3.841$, reject — the first predictor carries information.

   C against B: $190-184=6$ on $3$ degrees of freedom; $6<7.815$, do not reject — the three extra predictors, taken together, add nothing detectable beyond B.

   C against A: $200-184=16$ on $4$ degrees of freedom; $16>9.488$, reject — C is better than the empty model, but (from the previous test) that is because it contains B's predictor.

   (b) Nesting. Each model's parameter space contains the previous one's — set the new coefficients to zero and you recover the smaller model — so the maximised log-likelihood can only rise when parameters are added, and $D=-2\ell (\hat{\mathbf{\beta }})$ can only fall. The identity $10+6=16$ holds because all three statistics are differences of the same three deviances: $(D_A-D_B)+(D_B-D_C)=D_A-D_C$. The degrees of freedom add in the same way, $1+3=4$.

   (c) Wald versus likelihood ratio. $z=\frac{3/5}{1/5}=3$, so $z^2=9$, against the likelihood-ratio statistic $10$ for the same hypothesis ${\beta }_1=0$. Both are asymptotically ${\chi }_1^2$ under the null, but they are different functions of the data — one uses the curvature at the maximum, the other the height difference between two maxima — and they agree only in the limit. A gap of $9$ versus $10$ is unremarkable; both reject comfortably.

   (d) Build on the log-odds scale, then exponentiate the endpoints. The interval for ${\beta }_1$ is $\frac{3}{5}\pm 2\cdot \frac{1}{5}=\left(\frac{1}{5},1\right)$, and because $\exp$ is increasing the odds-ratio interval is $\left(e^{1/5},e\right)$ with the same coverage.

   The gaps on either side of the point estimate $e^{3/5}$ are

   $$e^{3/5}-e^{1/5}=e^{1/5}(e^{2/5}-1),e-e^{3/5}=e^{3/5}(e^{2/5}-1),$$

   so the upper gap is exactly $e^{2/5}$ times the lower one. The interval is symmetric on the log scale (the point estimate is its geometric centre, since $e^{1/5}\cdot e=e^{6/5}=(e^{3/5}{)}^2$) and skewed to the right on the odds-ratio scale, because $\exp$ is convex. That skew is a feature, not an error: the sampling distribution of $e^{{\hat{\beta }}_1}$ is itself right-skewed.

9. 9.Sketch · 9 min

   A continuous predictor is cut into five bins of $90$ observations each, with midpoints $x=1,2,3,4,5$; the success counts in the bins are $10,18,30,45,60$. A second predictor, cut into four bins with midpoints $x=1,2,3,4$, gives observed success proportions $\frac{1}{2},\frac{2}{3},\frac{8}{9},\frac{64}{65}$.

   A sketch with no numbers beside it earns nothing here: every plotted point must carry its exact coordinates.

   1. (a)For the first predictor, compute the empirical odds and log-odds in each bin exactly, and sketch the log-odds against $x$. Say what feature of the sketch supports linearity in the logit and what slope it suggests.
   2. (b)On a second set of axes, sketch the observed proportions against $x$ for the same predictor. Explain why this plot, which is visibly curved, is not evidence against the model.
   3. (c)Repeat (a) for the second predictor. Describe the curvature you find, identify its exact form, and say how the model should be amended.
   Show the solution to question 9

   (a) Odds first, then logs. The proportions are $\frac{10}{90}=\frac{1}{9},\frac{18}{90}=\frac{1}{5},\frac{30}{90}=\frac{1}{3},\frac{45}{90}=\frac{1}{2},\frac{60}{90}=\frac{2}{3}$. The odds $p/(1-p)$ are

   $$\frac{1}{8},\frac{1}{4},\frac{1}{2},1,2,$$

   and the log-odds are $-3\ln 2,-2\ln 2,-\ln 2,0,\ln 2$. Every successive difference is exactly $\ln 2$, so the five points are collinear.

   The sketch: horizontal axis $x$ from $0$ to $6$, vertical axis log-odds from $-2.5$ to $1$ (mark $\ln 2\approx 0.69$ as a tick). Plot the five points and draw the straight line through them; it crosses the axis at $(4,0)$ and has slope $\ln 2$. The feature that supports linearity in the logit is the constant vertical spacing — equal steps in $x$ give equal steps in log-odds — and the common step estimates ${\beta }_1\approx \ln 2$, with ${\beta }_0\approx -4\ln 2$ from the intercept of the line.

   (b) The same data on the probability scale. The points are $(1,\frac{1}{9}),(2,\frac{1}{5}),(3,\frac{1}{3}),(4,\frac{1}{2}),(5,\frac{2}{3})$; the successive rises are $\frac{4}{45},\frac{2}{15},\frac{1}{6},\frac{1}{6}$ — increasing, then levelling — which is the lower half of an S. Draw the axes with $p$ from $0$ to $1$, plot the five points, and draw through them the curve $p=\sigma ((x-4)\ln 2)$, which passes through all five exactly and flattens toward the ceiling $p=1$ (dashed) to the right.

   This curvature is not evidence against anything: the model asserts linearity in the logit, and a linear logit produces exactly an S-shaped probability curve. Checking straightness on the probability scale tests a model nobody proposed. That is why the diagnostic is made on the log-odds scale, where (a) shows a perfect line.

   (c) The second predictor is curved where it matters. Odds: $\frac{1/2}{1/2}=1,\frac{2/3}{1/3}=2,\frac{8/9}{1/9}=8,\frac{64/65}{1/65}=64$. Log-odds: $0,\ln 2,3\ln 2,6\ln 2$. The successive differences are $\ln 2,2\ln 2,3\ln 2$ — themselves increasing by a constant $\ln 2$ — so the log-odds are a quadratic in $x$: indeed $0,1,3,6$ are $\frac{x(x-1)}{2}$ at $x=1,2,3,4$, so

   $$\mathrm{logit}(p)=\frac{\ln 2}{2}x(x-1).$$

   Sketch the four points on axes with log-odds from $0$ to $4.5$; they bend upward, convex, with no straight line passing near all four (a line through the first two, slope $\ln 2$, would predict $2\ln 2$ at $x=3$ against the observed $3\ln 2$, and $3\ln 2$ at $x=4$ against $6\ln 2$). The remedy is the one ordinary regression would use: add an $x^2$ term, or a spline, so that the logit is linear in the transformed predictor. Here the quadratic term removes the curvature exactly.

10. 10.Computation · 10 min

    A fitted model is applied to seven cases. The three cases with $Y=1$ receive fitted probabilities $0.9,0.6,0.3$; the four cases with $Y=0$ receive $0.8,0.5,0.2,0.1$. A case is classified positive when its fitted probability exceeds the threshold $c$.

    1. (a)At $c=\frac{1}{2}$ write down $TP$, $FP$, $TN$, $FN$ and compute sensitivity, specificity, precision and accuracy. Repeat at $c=0.7$, and check the direction of every change against what the threshold trade-off requires.
    2. (b)Compute the AUC exactly, using its interpretation as a probability about a randomly chosen positive–negative pair.
    3. (c)List the point $(1-\text{specificity},\text{sensitivity})$ at every threshold at which the classification changes, as $c$ sweeps from $1$ down to $0$; join them into the ROC curve, and verify that the area under your polygon agrees with (b).
    4. (d)A second model returns exactly half of every fitted probability above. State its AUC and its confusion matrix at $c=\frac{1}{2}$. What does the comparison show about what AUC does and does not measure?
    Show the solution to question 10

    (a) Count, then divide. At $c=\frac{1}{2}$ the positives $0.9$ and $0.6$ are called positive and $0.3$ is not, so $TP=2$, $FN=1$; among the negatives only $0.8$ is called positive, so $FP=1$, $TN=3$. Then

    $$\text{sensitivity}=\frac{2}{3},\text{specificity}=\frac{3}{4},\text{precision}=\frac{2}{3},\text{accuracy}=\frac{5}{7}.$$

    At $c=0.7$ only $0.9$ and $0.8$ exceed the threshold: $TP=1$, $FN=2$, $FP=1$, $TN=3$, giving sensitivity $\frac{1}{3}$, specificity $\frac{3}{4}$, precision $\frac{1}{2}$, accuracy $\frac{4}{7}$.

    Raising $c$ moved one positive case ($0.6$) from predicted-positive to predicted-negative and moved no negative case, so sensitivity fell and specificity stayed put — non-increasing and non-decreasing respectively, exactly as the trade-off allows. Accuracy fell from $\frac{5}{7}$ to $\frac{4}{7}$; note the do-nothing baseline of calling everything negative already scores $\frac{4}{7}$.

    (b) Count concordant pairs. There are $3\times 4=12$ positive–negative pairs. The positive scored $0.9$ beats all four negatives; $0.6$ beats $0.5,0.2,0.1$ (three); $0.3$ beats $0.2,0.1$ (two). Nine of twelve pairs are ordered correctly, so

    $$\text{AUC}=\frac{9}{12}=\frac{3}{4}.$$

    There are no ties, so no half-credit is needed.

    (c) Walk down the sorted scores. In decreasing order the scores are $0.9(+),0.8(-),0.6(+),0.5(-),0.3(+),0.2(-),0.1(-)$. Start at $c=1$ with nothing called positive: the point is $(0,0)$. As $c$ drops past each score that case becomes positive; a positive case raises sensitivity by $\frac{1}{3}$, a negative case raises $1-\text{specificity}$ by $\frac{1}{4}$:

    $$(0,0)\to (0,\frac{1}{3})\to (\frac{1}{4},\frac{1}{3})\to (\frac{1}{4},\frac{2}{3})\to (\frac{1}{2},\frac{2}{3})\to (\frac{1}{2},1)\to (\frac{3}{4},1)\to (1,1).$$

    The ROC curve is this staircase: up, right, up, right, up, right, right. Draw it on the unit square with the chance diagonal dashed; the curve lies above the diagonal throughout.

    Area: on the four columns of width $\frac{1}{4}$ the staircase sits at heights $\frac{1}{3},\frac{2}{3},1,1$, so the area is $\frac{1}{4}\left(\frac{1}{3}+\frac{2}{3}+1+1\right)=\frac{1}{4}\cdot 3=\frac{3}{4}$ ✓. Rule the square into a $4\times 3$ grid of cells of area $\frac{1}{12}$: each cell corresponds to one (negative, positive) pair, and the cell lies under the curve exactly when the positive outscores the negative. Nine cells under the curve, nine concordant pairs — the two computations are the same count.

    (d) Halving preserves order. The second model's scores are $0.45,0.3,0.15$ for the positives and $0.4,0.25,0.1,0.05$ for the negatives; every pairwise comparison is unchanged, so its AUC is still $\frac{3}{4}$ and its ROC curve is identical. But at $c=\frac{1}{2}$ nothing exceeds the threshold: $TP=0$, $FN=3$, $FP=0$, $TN=4$, sensitivity $0$, specificity $1$, accuracy $\frac{4}{7}$, precision undefined ($\frac{0}{0}$). It never finds a single case.

    AUC measures ranking — whether positives tend to score above negatives — and is blind to the scale of the probabilities. Calibration, whether a case scored $0.45$ really has a $45\%$ chance, is a separate property, and the second model fails it at every case while matching the first model's AUC exactly. If the probabilities themselves will be used, or a threshold will be chosen from them, calibration must be checked separately.

11. 11.Proof · 10 min

    Four observations of the model $\mathrm{logit}(p_i)={\beta }_0+{\beta }_1{x}_i$ are $(x,y)=(1,0),(2,0),(3,1),(4,1)$. Write $\mathbf{\beta }=({\beta }_0,{\beta }_1{)}^{\top }$ and ${\mathbf{x}}_i=(1,x_i{)}^{\top }$.

    1. (a)Show that $\mathbf{b}=(-5,2{)}^{\top }$ completely separates the data.
    2. (b)Write $\ell (t\mathbf{b})$ explicitly as a function of $t>0$ and prove that it increases to $0$ as $t\to \infty$. Then prove that $\ell (\mathbf{\beta })<0$ for every finite $\mathbf{\beta }$, and conclude that the maximum likelihood estimate does not exist.
    3. (c)A fifth observation $(2,1)$ is added. Prove that no vector now separates the data, and show that $\ell (\mathbf{\beta })\le -2\ln 2$ for every $\mathbf{\beta }$, so that the supremum in (b) is no longer approachable.
    4. (d)Describe what software fitting the original four observations by Newton's method would report, and explain why running more iterations is not the remedy.
    Show the solution to question 11

    (a) Check the sign at every observation. ${\mathbf{x}}_i^{\top }\mathbf{b}=-5+2x_i$ takes the values $-3,-1,1,3$ at $x=1,2,3,4$: negative at both observations with $y=0$ and positive at both with $y=1$. That is the definition of complete separation.

    (b) Along the ray $t\mathbf{b}$. The linear predictors are ${\eta }_i=t(-5+2x_i)=-3t,-t,t,3t$. An observation with $y=0$ contributes $\ln (1-p_i)=-\ln (1+e^{{\eta }_i})$, and one with $y=1$ contributes $\ln p_i=-\ln (1+e^{-{\eta }_i})$. Substituting,

    $$\ell (t\mathbf{b})=-\ln (1+e^{-3t})-\ln (1+e^{-t})-\ln (1+e^{-t})-\ln (1+e^{-3t})=-2\ln (1+e^{-t})-2\ln (1+e^{-3t}).$$

    As $t$ increases, $e^{-t}$ and $e^{-3t}$ decrease to $0$, so each logarithm decreases to $\ln 1=0$ and $\ell (t\mathbf{b})$ increases. It is always negative, since each logarithm is of a number exceeding $1$; and using $\ln (1+u)\le u$ for $u\ge 0$,

    $$0>\ell (t\mathbf{b})\ge -2e^{-t}-2e^{-3t}⟶0,$$

    so $\ell (t\mathbf{b})\to 0$ from below.

    For any finite $\mathbf{\beta }$ every $p_i=\sigma ({\mathbf{x}}_i^{\top }\mathbf{\beta })$ lies strictly inside $(0,1)$, so every contribution $\ln p_i$ or $\ln (1-p_i)$ is strictly negative and $\ell (\mathbf{\beta })<0$. Combining: the supremum of $\ell$ is $0$, it is approached along the ray, and it is attained nowhere. A maximiser would have to attain the supremum, so none exists. $■$

    (c) Two outcomes at one predictor value. Separation would need a $\mathbf{b}$ with ${\mathbf{x}}^{\top }\mathbf{b}<0$ at $(2,0)$ and ${\mathbf{x}}^{\top }\mathbf{b}>0$ at $(2,1)$, but both observations share $\mathbf{x}=(1,2{)}^{\top }$, so the same number would have to be both negative and positive. No separating vector exists.

    For the bound: the two observations at $x=2$ share a fitted probability $p$, and together they contribute

    $$\ln (1-p)+\ln p=\ln (p(1-p))\le \ln \frac{1}{4}=-2\ln 2,$$

    since $p(1-p)\le \frac{1}{4}$ for $p\in (0,1)$. The other three contributions are negative, so $\ell (\mathbf{\beta })\le -2\ln 2$ for every $\mathbf{\beta }$. The likelihood can no longer be driven towards $0$ by any choice of coefficients: the conflicting pair at $x=2$ costs at least $2\ln 2$ whatever the fit does elsewhere. $■$

    (d) What the output looks like. Each Newton iteration moves further out along a separating direction, so the reported coefficients grow with the number of iterations — a slope in the tens or twenties, an intercept of the opposite sign and comparable size — and the iteration halts only when the change in $\ell$ falls below the tolerance, since $\ell$ is flattening towards $0$. The fitted probabilities are numerically $0$ and $1$, and the weights $p_i(1-p_i)$ are numerically $0$, so ${\mathbf{X}}^{\top }\mathbf{WX}$ is nearly singular and its inverse — the standard errors — is enormous. Software may add a warning about fitted probabilities of $0$ or $1$.

    More iterations cannot help because there is nothing to converge to: (b) shows the maximum does not exist, so any finite stopping point is arbitrary. The remedies are to change the problem — penalise the likelihood (Firth's correction or a ridge penalty, which make the penalised objective attain a finite maximum), drop or merge the offending predictor, or collect data that overlaps as in (c).

12. 12.Critique · 9 min

    Each of the three modelling decisions below contains an error. Identify it precisely, explain why it matters, and state what should have been done. Where a number is involved, correct it exactly.

    1. (a)“A Poisson regression of monthly complaints per store has coefficient $\ln 2$ on the number of checkouts. So each additional checkout adds $\ln 2\approx 0.69$ complaints to the expected monthly count.”
    2. (b)“Across stores the variance of the monthly complaint count is about six times its mean. Since over-dispersion leaves the Poisson coefficients roughly unbiased, the reported standard errors and $p$-values can be used as they are. In particular the checkout coefficient has $z=3$, so it is significant.”
    3. (c)“Time spent on a web page, in minutes, is positive and right-skewed, so we fitted a Poisson regression with a log link: the log link guarantees positive predictions, and the skew is handled by the distribution.”
    Show the solution to question 12

    (a) The log link makes coefficients multiplicative. In the model $\log E(Y)=\eta$ the expected count is $e^{\eta }$, and increasing the checkout count by one replaces $\eta$ by $\eta +\ln 2$, so it multiplies the expected count by $e^{\ln 2}=2$. The correct sentence is “each additional checkout doubles the expected number of complaints”. The additive change is not $\ln 2$ but $E(Y)$ itself: from an expected $3$ complaints the extra checkout adds $3$, from $10$ it adds $10$. This is the same misreading as treating a logistic coefficient as a change in probability — the coefficient lives on the scale of the link, and only there is its effect constant.

    (b) Unbiased coefficients do not rescue the inference. A Poisson response has variance equal to its mean, and the standard errors are computed on that assumption. If the true variance is about $6$ times the mean, the actual sampling variance of $\hat{\mathbf{\beta }}$ is about $6$ times what the model claims, so every reported standard error is too small by a factor of about $\sqrt{6}$ and every $z$-statistic too large by the same factor. The corrected statistic is

    $$z=\frac{3}{\sqrt{6}}=\frac{\sqrt{6}}{2}\approx 1.22,$$

    well short of $2$: the checkout effect is not significant at the $5\%$ level once the dispersion is honoured. This is the count-data twin of heteroscedasticity in the linear model — estimates fine, standard errors wrong. The remedy is a negative binomial regression, which estimates a dispersion parameter, or at least a quasi-Poisson or robust standard error that scales the Poisson ones by $\sqrt{\hat{\varphi }}$.

    (c) The link is right and the distribution is wrong. The Poisson distribution lives on the counts $0,1,2,\dots$; time on a page is continuous, and a model for it that assigns probability only to whole minutes is not a model of the data. The mistake shows up concretely through the variance–mean relation, which is not invariant to the unit of measurement: a page with mean $2$ minutes and Poisson-consistent variance $2$ (min${}^2$) has, in seconds, mean $120$ and variance $2\times 60^2=7200$, not $120$. A model whose fit depends on whether you recorded seconds or minutes is measuring the units, not the behaviour.

    The positive-predictions argument is correct as far as it goes — a log link does keep $E(Y)>0$ — but link and distribution are separate choices in a generalised linear model and the second was made badly. The right choices are ordinary regression on $\ln (\text{time})$, or a gamma GLM with a log link, whose variance is proportional to the square of the mean: mean $2$, variance $4\varphi$ in minutes becomes mean $120$, variance $14400\varphi =\varphi \cdot 120^2$ in seconds — the same model in both units, as a model for a continuous quantity must be.