Logistic Regression

(a) Solve the two inequalities. $\hat{p}\ge 0$ requires $\frac{x}{20}\ge \frac{1}{5}$, that is $x\ge 4$; and $\hat{p}\le 1$ requires $\frac{x}{20}\le \frac{6}{5}$, that is $x\le 24$. The fit is a probability only for incomes between $\$4,000$ and $\$24,000$.

At $x=2$: $\hat{p}=-\frac{1}{5}+\frac{1}{10}=-\frac{1}{10}$, a negative probability. At $x=30$: $\hat{p}=-\frac{1}{5}+\frac{3}{2}=\frac{13}{10}$, a probability of $130\%$. Both are statements that cannot be true of any household.

(b) The error is two-valued. At $x=8$ the model probability is $p=-\frac{1}{5}+\frac{8}{20}=\frac{1}{5}$. Since $Y$ is Bernoulli$(p)$, the error $\epsilon =Y-p$ equals $1-p=\frac{4}{5}$ with probability $p=\frac{1}{5}$, and equals $-p=-\frac{1}{5}$ with probability $1-p=\frac{4}{5}$. Its variance is

$$p(1-p)=\frac{1}{5}\cdot \frac{4}{5}=\frac{4}{25}.$$

Check directly: $E(\epsilon )=\frac{4}{5}\cdot \frac{1}{5}-\frac{1}{5}\cdot \frac{4}{5}=0$ and $E({\epsilon }^2)=\frac{16}{25}\cdot \frac{1}{5}+\frac{1}{25}\cdot \frac{4}{5}=\frac{20}{125}=\frac{4}{25}$. ✓

At $x=14$: $p=-\frac{1}{5}+\frac{14}{20}=\frac{1}{2}$. The error is $\pm \frac{1}{2}$, each with probability $\frac{1}{2}$, and the variance is $\frac{1}{2}\cdot \frac{1}{2}=\frac{1}{4}$.

(c) Rank the failures. Part (a) is the range failure: the fitted line leaves $[0,1]$. Part (b) shows the other two at once. The variance is $\frac{4}{25}$ at one income and $\frac{1}{4}$ at another, so the errors are heteroscedastic; and at each income the error takes only two values, so it is not normal.

Heteroscedasticity and non-normality are the familiar complaints: by Gauss–Markov the least-squares estimates stay unbiased, only the standard errors are wrong, and weighted least squares or robust standard errors repair them. The range failure is different in kind. A fitted probability of $\frac{13}{10}$ is not an imprecise estimate of something — the quantity being estimated cannot exceed $1$ — so no confidence interval around it helps. That failure forces a new functional form, which is why the log-odds is modelled instead of the probability.

(a) Symmetry, then the derivative. By the formula, $\sigma (-z)=\frac{1}{1+e^z}$. Multiply numerator and denominator by $e^{-z}$:

$$\sigma (-z)=\frac{e^{-z}}{e^{-z}+1}.$$

On the other side, $1-\sigma (z)=\frac{(1+e^{-z})-1}{1+e^{-z}}=\frac{e^{-z}}{1+e^{-z}}$, the same expression. So $\sigma (-z)=1-\sigma (z)$.

For the derivative, write $\sigma (z)=(1+e^{-z}{)}^{-1}$ and apply the chain rule:

$${\sigma }^{\prime }(z)=-(1+e^{-z}{)}^{-2}\cdot (-e^{-z})=\frac{1}{1+e^{-z}}\cdot \frac{e^{-z}}{1+e^{-z}}=\sigma (z)\sigma (-z),$$

using the expression for $\sigma (-z)$ just found. Equivalently ${\sigma }^{\prime }=\sigma (1-\sigma )$.

(b) Complete the square. Put $p=\sigma (z)$. Since $e^{-z}>0$, the denominator exceeds $1$, so $0<p<1$. Then

$${\sigma }^{\prime }(z)=p(1-p)=\frac{1}{4}-{\left(p-\frac{1}{2}\right)}^2\le \frac{1}{4},$$

with equality exactly when $p=\frac{1}{2}$. Now $\sigma (z)=\frac{1}{2}$ means $1+e^{-z}=2$, so $e^{-z}=1$ and $z=0$. Both factors of ${\sigma }^{\prime }$ are positive, so $\sigma$ is strictly increasing and takes the value $\frac{1}{2}$ only once, at $z=0$. Hence the bound $\frac{1}{4}$ is attained only there.

(c) Differentiate ${\sigma }^{\prime }=\sigma -{\sigma }^2$. By the product/chain rule,

$${\sigma }^{\prime \prime }={\sigma }^{\prime }-2\sigma {\sigma }^{\prime }={\sigma }^{\prime }(1-2\sigma ).$$

Since ${\sigma }^{\prime }>0$ everywhere, the sign of ${\sigma }^{\prime \prime }$ is the sign of $1-2\sigma (z)$. For $z<0$ we have $\sigma (z)<\frac{1}{2}$ (strictly increasing, $\sigma (0)=\frac{1}{2}$), so ${\sigma }^{\prime \prime }>0$ and $\sigma$ is convex there; for $z>0$, ${\sigma }^{\prime \prime }<0$ and $\sigma$ is concave. The sign changes at $z=0$, so the inflection point is $\left(0,\frac{1}{2}\right)$ — the same point at which, by (b), the slope is largest.

For a logistic model this says: a unit change in a predictor moves the probability most when the current probability is near $\frac{1}{2}$, and the effect shrinks towards zero as the probability approaches $0$ or $1$. The steepest possible probability change per unit of log-odds is $\frac{1}{4}$, and it is only available at the centre. $■$

(a) Work on the odds scale first. The linear predictor is $\eta =(x-2)\ln 3$, so the odds are

$$\frac{\hat{p}}{1-\hat{p}}=e^{\eta }=e^{(x-2)\ln 3}=3^{x-2}.$$

At $x=0,1,2,3,4$ this gives odds $\frac{1}{9},\frac{1}{3},1,3,9$. Converting with $\hat{p}=\frac{\text{odds}}{1+\text{odds}}$:

$$\hat{p}=\frac{1}{10},\frac{1}{4},\frac{1}{2},\frac{3}{4},\frac{9}{10}.$$

(For instance at $x=1$: $\frac{1/3}{1+1/3}=\frac{1/3}{4/3}=\frac{1}{4}$.)

(b) Coefficients are log odds ratios. A $1$ mm increase adds ${\beta }_1=\ln 3$ to the log-odds, so it multiplies the odds by $e^{\ln 3}=3$; a $2$ mm increase adds $2\ln 3$ and multiplies them by $e^{2\ln 3}=9$. The reason it is independent of the starting point is that the odds are $e^{\eta }$ and $e^{\eta +{\beta }_1}=e^{\eta }{e}^{{\beta }_1}$: the ratio of new odds to old is $e^{{\beta }_1}$ whatever $\eta$ was. The table in (a) shows it — $\frac{1}{9}\to \frac{1}{3}\to 1\to 3\to 9$ is a factor of $3$ at every step.

(c) The probability steps are not constant.

$$\frac{1}{2}-\frac{1}{4}=\frac{1}{4},\frac{3}{4}-\frac{1}{2}=\frac{1}{4},\frac{9}{10}-\frac{3}{4}=\frac{3}{20}.$$

The first two steps are equal because they are mirror images about $p=\frac{1}{2}$: $\sigma (-z)=1-\sigma (z)$ places $\frac{1}{4}$ and $\frac{3}{4}$ symmetrically. The third step is smaller because the curve is flattening as the probability nears $1$ — the same $\ln 3$ on the log-odds scale buys less and less probability. There is no contradiction with (b): the odds ratio is exactly $3$ at each step, and it is the probability that has no fixed exchange rate against the coefficient.

(a) The relative risk is $\text{RR}=\frac{p_1}{p_2}=\frac{1/2}{1/4}=2$. The odds are $\frac{1/2}{1/2}=1$ for treated and $\frac{1/4}{3/4}=\frac{1}{3}$ for control, so $\text{OR}=\frac{1}{1/3}=3$. Treatment doubles the risk but triples the odds.

A logistic coefficient is a log odds ratio, never a log relative risk: with treatment coded $x=1$, ${\beta }_1=\ln \text{OR}=\ln 3$.

(b) $\text{RR}=\frac{2/100}{1/100}=2$ again. The odds are $\frac{2/100}{98/100}=\frac{2}{98}=\frac{1}{49}$ and $\frac{1/100}{99/100}=\frac{1}{99}$, so

$$\text{OR}=\frac{1/49}{1/99}=\frac{99}{49}\approx \mathrm{2.02.}$$

For a rare outcome the two measures nearly agree; for the common outcome in (a) they did not.

(c) Rearrange the definition.

$$\text{OR}=\frac{p_1/(1-p_1)}{p_2/(1-p_2)}=\frac{p_1}{p_2}\cdot \frac{1-p_2}{1-p_1}=\text{RR}\cdot \frac{1-p_2}{1-p_1}.$$

If $p_1>p_2$ then $1-p_2>1-p_1>0$, so the correction factor exceeds $1$ and $\text{OR}>\text{RR}$: an odds ratio reported in the language of risk always overstates a beneficial-looking effect. The factor $\frac{1-p_2}{1-p_1}$ tends to $1$ as $p_1,p_2\to 0$, which is the rare-outcome case in which the two agree.

Checks: in (a) the factor is $\frac{3/4}{1/2}=\frac{3}{2}$ and $2\cdot \frac{3}{2}=3$ ✓; in (b) it is $\frac{99/100}{98/100}=\frac{99}{98}$ and $2\cdot \frac{99}{98}=\frac{99}{49}$ ✓.

(a) A coefficient is a change in log-odds, not in probability. The correct multiplicative statement is: each extra year multiplies the odds of default by $e^{-1/2}$, and five years multiply them by $e^{-5/2}$. The probability change has no fixed value — it depends on where you start — and it can be bounded. By the mean value theorem, for some $\xi$ between $\eta$ and $\eta -\frac{1}{2}$,

$$\sigma (\eta )-\sigma \left(\eta -\frac{1}{2}\right)=\frac{1}{2}{\sigma }^{\prime }(\xi )\le \frac{1}{2}\cdot \frac{1}{4}=\frac{1}{8},$$

since ${\sigma }^{\prime }=\sigma (1-\sigma )\le \frac{1}{4}$. So no single year can move the probability by more than $\frac{1}{8}$, let alone $\frac{1}{2}$. For instance from $p=\frac{1}{2}$ (where $\eta =0$) one more year gives $p=\sigma (-\frac{1}{2})=\frac{1}{1+e^{1/2}}$, which exceeds $\frac{1}{3}$ because $e^{1/2}<2$: a drop of less than $\frac{1}{6}$. And the probability can never reach zero, since $\sigma$ takes values in $(0,1)$; it only approaches it.

(b) The interval is built correctly and transformed incorrectly. On the log-odds scale ${\hat{\beta }}_1$ is approximately normal, so $\left(-\frac{9}{10},-\frac{1}{10}\right)$ is right. To move to the odds-ratio scale, apply $\exp$ to the two endpoints: because $\exp$ is strictly increasing, ${\beta }_1$ lies in the interval exactly when $e^{{\beta }_1}$ lies in $\left(e^{-9/10},e^{-1/10}\right)$, so the coverage is unchanged.

The proposed interval $e^{-1/2}\pm e^{2/5}$ is not even a set of odds ratios: its lower end is $e^{-1/2}-e^{2/5}<0$, since $e^{2/5}>1>e^{-1/2}$, and an odds ratio is positive. The correct interval is not symmetric about $e^{-1/2}$; rather $e^{-9/10}\cdot e^{-1/10}=e^{-1}={\left(e^{-1/2}\right)}^2$, so $e^{-1/2}$ is its geometric centre. Consistency check: the Wald statistic is $z=\frac{-1/2}{1/5}=-\frac{5}{2}$, with $|z|>2$; the log-odds interval excludes $0$ and the odds-ratio interval excludes $1$, as they must.

(c) An odds ratio is not a relative risk. “Less likely” is a statement about probabilities, and by the identity $\text{OR}=\text{RR}\cdot \frac{1-p_2}{1-p_1}$ the two agree only when default is rare. The correct sentence is “one more year multiplies the odds of default by $e^{-1/2}$”. To see the size of the discrepancy, take a borrower whose current probability is $\frac{1}{2}$: after one more year the odds are $e^{-1/2}$, so the probability is $\frac{1}{1+e^{1/2}}$ and the relative risk is

$$\text{RR}=\frac{2}{1+e^{1/2}}>e^{-1/2},$$

because $2e^{1/2}>1+e^{1/2}$ reduces to $e^{1/2}>1$. The risk falls by less than $39\%$; the odds fall by $39\%$. Only for a borrower whose default probability is already tiny would the two readings coincide.

(a) Sum the Bernoulli contributions. With every linear predictor equal to $\beta$,

$$\ell (\beta )=\sum _{i=1}^n[y_i\beta -\ln (1+e^{\beta }\left)\right]=s\beta -n\ln (1+e^{\beta }).$$

Differentiating, ${\ell }^{\prime }(\beta )=s-n\frac{e^{\beta }}{1+e^{\beta }}=s-n\sigma (\beta )$, which is the score equation ${\sum }_i(y_i-p_i)=0$ with all $p_i$ equal. Setting it to zero gives $\sigma (\hat{\beta })=\frac{s}{n}=\frac{3}{4}$, so $\hat{\beta }=\mathrm{logit}\left(\frac{3}{4}\right)=\ln \frac{3/4}{1/4}=\ln 3$. Because $\ell$ is strictly concave (its second derivative $-n\sigma (1-\sigma )$ is negative), this stationary point is the unique global maximum.

(b) One Newton step. At ${\beta }^{(0)}=0$ every $p_i=\sigma (0)=\frac{1}{2}$. The gradient is $s-n\cdot \frac{1}{2}=9-6=3$ and the Hessian is $-n\sigma (1-\sigma )=-12\cdot \frac{1}{4}=-3$. Newton's update is

$${\beta }^{(1)}={\beta }^{(0)}-\frac{{\ell }^{\prime }(0)}{{\ell }^{\prime \prime }(0)}=0-\frac{3}{-3}=1.$$

The exact maximiser is $\ln 3$, and $\ln 3>1$ because $3>e$. So the step undershoots, landing at $1$ against the target $\ln 3\approx 1.099$ — most of the way in a single step, as expected near the optimum of a concave function; a second step would close almost all of the remaining gap.

(c) Curvature at the maximiser. With $\sigma (\hat{\beta })=\frac{3}{4}$,

$${\ell }^{\prime \prime }(\hat{\beta })=-n\cdot \frac{3}{4}\cdot \frac{1}{4}=-\frac{9}{4}.$$

The observed information is $\frac{9}{4}$, and the standard error is its inverse square root: $SE(\hat{\beta })=\sqrt{\frac{4}{9}}=\frac{2}{3}$. (In general $SE=1/\sqrt{n\hat{p}(1-\hat{p})}$, the familiar binomial standard error transported to the log-odds scale.)

The Wald statistic is $z=\frac{\hat{\beta }}{SE}=\frac{\ln 3}{2/3}=\frac{3}{2}\ln 3$. It exceeds $2$ exactly when $\ln 3>\frac{4}{3}$, that is when $3>e^{4/3}\approx 3.79$, which is false. So $z\approx 1.65<2$ and we do not reject $\beta =0$ at the $5\%$ level: nine successes in twelve trials is consistent with a fair coin.

(d) Deviance. At $\hat{\beta }=\ln 3$ we have $1+e^{\hat{\beta }}=4$, so

$$\ell (\hat{\beta })=9\ln 3-12\ln 4=9\ln 3-24\ln 2,D=-2\ell (\hat{\beta })=48\ln 2-18\ln 3.$$

Check by the other route: for an intercept-only fit $\ell (\hat{\beta })=s\ln \frac{s}{n}+(n-s)\ln \frac{n-s}{n}=9\ln \frac{3}{4}+3\ln \frac{1}{4}=9\ln 3-18\ln 2-6\ln 2=9\ln 3-24\ln 2$ ✓. Numerically $D\approx 33.27-19.78=13.5$.

(a) Two columns, two equations. The design matrix has a column of ones and the indicator column $(x_i)$. The score equations ${\mathbf{X}}^{\top }(\mathbf{y}-\hat{\mathbf{p}})=0$ are therefore

$$\sum _{i=1}^n(y_i-{\hat{p}}_i)=0\text{and}\sum _{i=1}^n{x}_i(y_i-{\hat{p}}_i)=\sum _{i:x_i=1}(y_i-{\hat{p}}_i)=0.$$

The second says ${\sum }_{x_i=1}{\hat{p}}_i={\sum }_{x_i=1}{y}_i=s_1$. Subtracting it from the first leaves ${\sum }_{x_i=0}(y_i-{\hat{p}}_i)=0$, so ${\sum }_{x_i=0}{\hat{p}}_i=s_0$. In words: with an intercept in the model, the fitted probabilities sum to the observed successes overall, and with an indicator in the model they do so within the indicated group as well.

(b) Same predictor, same fitted probability. Every observation with $x_i=1$ has the same linear predictor ${\hat{\beta }}_0+{\hat{\beta }}_1$, hence the same fitted probability, call it ${\hat{p}}_{(1)}$; every observation with $x_i=0$ has ${\hat{p}}_{(0)}=\sigma ({\hat{\beta }}_0)$. The sums in (a) become $n_1{\hat{p}}_{(1)}=s_1$ and $n_0{\hat{p}}_{(0)}=s_0$, so

$${\hat{p}}_{(1)}=\frac{s_1}{n_1},{\hat{p}}_{(0)}=\frac{s_0}{n_0}.$$

Applying the logit, ${\hat{\beta }}_0=\mathrm{logit}\left(\frac{s_0}{n_0}\right)=\ln \frac{s_0}{n_0-s_0}$ and ${\hat{\beta }}_0+{\hat{\beta }}_1=\ln \frac{s_1}{n_1-s_1}$. Subtracting,

$${\hat{\beta }}_1=\ln \frac{s_1}{n_1-s_1}-\ln \frac{s_0}{n_0-s_0}=\ln \frac{s_1/(n_1-s_1)}{s_0/(n_0-s_0)},$$

the log of the sample odds ratio. This stationary point is the maximum: both groups are present, so $\mathbf{X}$ has full rank, and the assumption $0<s_j<n_j$ places every fitted probability strictly inside $(0,1)$, so the log-likelihood is strictly concave and its unique stationary point is the global maximiser. $■$

(c) Numbers. ${\hat{p}}_{(1)}=\frac{30}{40}=\frac{3}{4}$ with odds $3$; ${\hat{p}}_{(0)}=\frac{15}{60}=\frac{1}{4}$ with odds $\frac{1}{3}$. Hence

$${\hat{\beta }}_0=\ln \frac{1}{3}=-\ln 3,{\hat{\beta }}_1=\ln 3-(-\ln 3)=2\ln 3=\ln 9,$$

an odds ratio of $9$ for the drug. Check: ${\hat{\beta }}_0+{\hat{\beta }}_1=\ln 3=\mathrm{logit}(\frac{3}{4})$ ✓.

If $s_1=n_1$ — every treated patient recovers — then (a) demands ${\hat{p}}_{(1)}=1$, which no finite ${\hat{\beta }}_0+{\hat{\beta }}_1$ can deliver because $\sigma$ never reaches $1$. The score equations then have no solution: this is separation on the indicator, the likelihood increases without bound as ${\beta }_1\to \infty$, and the maximum likelihood estimate does not exist. The sample odds ratio $\frac{n_1/0}{\cdots }$ is infinite for the same reason.

(a) Likelihood-ratio tests. For nested models differing by $k$ parameters, $D_{\text{red}}-D_{\text{full}}\to {\chi }_k^2$ under the null that the extra coefficients are zero — Wilks' theorem, quoted for twice the log-likelihood ratio.

B against A: $200-190=10$ on $1$ degree of freedom; $10>3.841$, reject — the first predictor carries information.

C against B: $190-184=6$ on $3$ degrees of freedom; $6<7.815$, do not reject — the three extra predictors, taken together, add nothing detectable beyond B.

C against A: $200-184=16$ on $4$ degrees of freedom; $16>9.488$, reject — C is better than the empty model, but (from the previous test) that is because it contains B's predictor.

(b) Nesting. Each model's parameter space contains the previous one's — set the new coefficients to zero and you recover the smaller model — so the maximised log-likelihood can only rise when parameters are added, and $D=-2\ell (\hat{\mathbf{\beta }})$ can only fall. The identity $10+6=16$ holds because all three statistics are differences of the same three deviances: $(D_A-D_B)+(D_B-D_C)=D_A-D_C$. The degrees of freedom add in the same way, $1+3=4$.

(c) Wald versus likelihood ratio. $z=\frac{3/5}{1/5}=3$, so $z^2=9$, against the likelihood-ratio statistic $10$ for the same hypothesis ${\beta }_1=0$. Both are asymptotically ${\chi }_1^2$ under the null, but they are different functions of the data — one uses the curvature at the maximum, the other the height difference between two maxima — and they agree only in the limit. A gap of $9$ versus $10$ is unremarkable; both reject comfortably.

(d) Build on the log-odds scale, then exponentiate the endpoints. The interval for ${\beta }_1$ is $\frac{3}{5}\pm 2\cdot \frac{1}{5}=\left(\frac{1}{5},1\right)$, and because $\exp$ is increasing the odds-ratio interval is $\left(e^{1/5},e\right)$ with the same coverage.

The gaps on either side of the point estimate $e^{3/5}$ are

$$e^{3/5}-e^{1/5}=e^{1/5}(e^{2/5}-1),e-e^{3/5}=e^{3/5}(e^{2/5}-1),$$

so the upper gap is exactly $e^{2/5}$ times the lower one. The interval is symmetric on the log scale (the point estimate is its geometric centre, since $e^{1/5}\cdot e=e^{6/5}=(e^{3/5}{)}^2$) and skewed to the right on the odds-ratio scale, because $\exp$ is convex. That skew is a feature, not an error: the sampling distribution of $e^{{\hat{\beta }}_1}$ is itself right-skewed.

(a) Odds first, then logs. The proportions are $\frac{10}{90}=\frac{1}{9},\frac{18}{90}=\frac{1}{5},\frac{30}{90}=\frac{1}{3},\frac{45}{90}=\frac{1}{2},\frac{60}{90}=\frac{2}{3}$. The odds $p/(1-p)$ are

$$\frac{1}{8},\frac{1}{4},\frac{1}{2},1,2,$$

and the log-odds are $-3\ln 2,-2\ln 2,-\ln 2,0,\ln 2$. Every successive difference is exactly $\ln 2$, so the five points are collinear.

The sketch: horizontal axis $x$ from $0$ to $6$, vertical axis log-odds from $-2.5$ to $1$ (mark $\ln 2\approx 0.69$ as a tick). Plot the five points and draw the straight line through them; it crosses the axis at $(4,0)$ and has slope $\ln 2$. The feature that supports linearity in the logit is the constant vertical spacing — equal steps in $x$ give equal steps in log-odds — and the common step estimates ${\beta }_1\approx \ln 2$, with ${\beta }_0\approx -4\ln 2$ from the intercept of the line.

(b) The same data on the probability scale. The points are $(1,\frac{1}{9}),(2,\frac{1}{5}),(3,\frac{1}{3}),(4,\frac{1}{2}),(5,\frac{2}{3})$; the successive rises are $\frac{4}{45},\frac{2}{15},\frac{1}{6},\frac{1}{6}$ — increasing, then levelling — which is the lower half of an S. Draw the axes with $p$ from $0$ to $1$, plot the five points, and draw through them the curve $p=\sigma ((x-4)\ln 2)$, which passes through all five exactly and flattens toward the ceiling $p=1$ (dashed) to the right.

This curvature is not evidence against anything: the model asserts linearity in the logit, and a linear logit produces exactly an S-shaped probability curve. Checking straightness on the probability scale tests a model nobody proposed. That is why the diagnostic is made on the log-odds scale, where (a) shows a perfect line.

(c) The second predictor is curved where it matters. Odds: $\frac{1/2}{1/2}=1,\frac{2/3}{1/3}=2,\frac{8/9}{1/9}=8,\frac{64/65}{1/65}=64$. Log-odds: $0,\ln 2,3\ln 2,6\ln 2$. The successive differences are $\ln 2,2\ln 2,3\ln 2$ — themselves increasing by a constant $\ln 2$ — so the log-odds are a quadratic in $x$: indeed $0,1,3,6$ are $\frac{x(x-1)}{2}$ at $x=1,2,3,4$, so

$$\mathrm{logit}(p)=\frac{\ln 2}{2}x(x-1).$$

Sketch the four points on axes with log-odds from $0$ to $4.5$; they bend upward, convex, with no straight line passing near all four (a line through the first two, slope $\ln 2$, would predict $2\ln 2$ at $x=3$ against the observed $3\ln 2$, and $3\ln 2$ at $x=4$ against $6\ln 2$). The remedy is the one ordinary regression would use: add an $x^2$ term, or a spline, so that the logit is linear in the transformed predictor. Here the quadratic term removes the curvature exactly.

(a) Count, then divide. At $c=\frac{1}{2}$ the positives $0.9$ and $0.6$ are called positive and $0.3$ is not, so $TP=2$, $FN=1$; among the negatives only $0.8$ is called positive, so $FP=1$, $TN=3$. Then

$$\text{sensitivity}=\frac{2}{3},\text{specificity}=\frac{3}{4},\text{precision}=\frac{2}{3},\text{accuracy}=\frac{5}{7}.$$

At $c=0.7$ only $0.9$ and $0.8$ exceed the threshold: $TP=1$, $FN=2$, $FP=1$, $TN=3$, giving sensitivity $\frac{1}{3}$, specificity $\frac{3}{4}$, precision $\frac{1}{2}$, accuracy $\frac{4}{7}$.

Raising $c$ moved one positive case ($0.6$) from predicted-positive to predicted-negative and moved no negative case, so sensitivity fell and specificity stayed put — non-increasing and non-decreasing respectively, exactly as the trade-off allows. Accuracy fell from $\frac{5}{7}$ to $\frac{4}{7}$; note the do-nothing baseline of calling everything negative already scores $\frac{4}{7}$.

(b) Count concordant pairs. There are $3\times 4=12$ positive–negative pairs. The positive scored $0.9$ beats all four negatives; $0.6$ beats $0.5,0.2,0.1$ (three); $0.3$ beats $0.2,0.1$ (two). Nine of twelve pairs are ordered correctly, so

$$\text{AUC}=\frac{9}{12}=\frac{3}{4}.$$

There are no ties, so no half-credit is needed.

(c) Walk down the sorted scores. In decreasing order the scores are $0.9(+),0.8(-),0.6(+),0.5(-),0.3(+),0.2(-),0.1(-)$. Start at $c=1$ with nothing called positive: the point is $(0,0)$. As $c$ drops past each score that case becomes positive; a positive case raises sensitivity by $\frac{1}{3}$, a negative case raises $1-\text{specificity}$ by $\frac{1}{4}$:

$$(0,0)\to (0,\frac{1}{3})\to (\frac{1}{4},\frac{1}{3})\to (\frac{1}{4},\frac{2}{3})\to (\frac{1}{2},\frac{2}{3})\to (\frac{1}{2},1)\to (\frac{3}{4},1)\to (1,1).$$

The ROC curve is this staircase: up, right, up, right, up, right, right. Draw it on the unit square with the chance diagonal dashed; the curve lies above the diagonal throughout.

Area: on the four columns of width $\frac{1}{4}$ the staircase sits at heights $\frac{1}{3},\frac{2}{3},1,1$, so the area is $\frac{1}{4}\left(\frac{1}{3}+\frac{2}{3}+1+1\right)=\frac{1}{4}\cdot 3=\frac{3}{4}$ ✓. Rule the square into a $4\times 3$ grid of cells of area $\frac{1}{12}$: each cell corresponds to one (negative, positive) pair, and the cell lies under the curve exactly when the positive outscores the negative. Nine cells under the curve, nine concordant pairs — the two computations are the same count.

(d) Halving preserves order. The second model's scores are $0.45,0.3,0.15$ for the positives and $0.4,0.25,0.1,0.05$ for the negatives; every pairwise comparison is unchanged, so its AUC is still $\frac{3}{4}$ and its ROC curve is identical. But at $c=\frac{1}{2}$ nothing exceeds the threshold: $TP=0$, $FN=3$, $FP=0$, $TN=4$, sensitivity $0$, specificity $1$, accuracy $\frac{4}{7}$, precision undefined ($\frac{0}{0}$). It never finds a single case.

AUC measures ranking — whether positives tend to score above negatives — and is blind to the scale of the probabilities. Calibration, whether a case scored $0.45$ really has a $45\%$ chance, is a separate property, and the second model fails it at every case while matching the first model's AUC exactly. If the probabilities themselves will be used, or a threshold will be chosen from them, calibration must be checked separately.

(a) Check the sign at every observation. ${\mathbf{x}}_i^{\top }\mathbf{b}=-5+2x_i$ takes the values $-3,-1,1,3$ at $x=1,2,3,4$: negative at both observations with $y=0$ and positive at both with $y=1$. That is the definition of complete separation.

(b) Along the ray $t\mathbf{b}$. The linear predictors are ${\eta }_i=t(-5+2x_i)=-3t,-t,t,3t$. An observation with $y=0$ contributes $\ln (1-p_i)=-\ln (1+e^{{\eta }_i})$, and one with $y=1$ contributes $\ln p_i=-\ln (1+e^{-{\eta }_i})$. Substituting,

$$\ell (t\mathbf{b})=-\ln (1+e^{-3t})-\ln (1+e^{-t})-\ln (1+e^{-t})-\ln (1+e^{-3t})=-2\ln (1+e^{-t})-2\ln (1+e^{-3t}).$$

As $t$ increases, $e^{-t}$ and $e^{-3t}$ decrease to $0$, so each logarithm decreases to $\ln 1=0$ and $\ell (t\mathbf{b})$ increases. It is always negative, since each logarithm is of a number exceeding $1$; and using $\ln (1+u)\le u$ for $u\ge 0$,

$$0>\ell (t\mathbf{b})\ge -2e^{-t}-2e^{-3t}⟶0,$$

so $\ell (t\mathbf{b})\to 0$ from below.

For any finite $\mathbf{\beta }$ every $p_i=\sigma ({\mathbf{x}}_i^{\top }\mathbf{\beta })$ lies strictly inside $(0,1)$, so every contribution $\ln p_i$ or $\ln (1-p_i)$ is strictly negative and $\ell (\mathbf{\beta })<0$. Combining: the supremum of $\ell$ is $0$, it is approached along the ray, and it is attained nowhere. A maximiser would have to attain the supremum, so none exists. $■$

(c) Two outcomes at one predictor value. Separation would need a $\mathbf{b}$ with ${\mathbf{x}}^{\top }\mathbf{b}<0$ at $(2,0)$ and ${\mathbf{x}}^{\top }\mathbf{b}>0$ at $(2,1)$, but both observations share $\mathbf{x}=(1,2{)}^{\top }$, so the same number would have to be both negative and positive. No separating vector exists.

For the bound: the two observations at $x=2$ share a fitted probability $p$, and together they contribute

$$\ln (1-p)+\ln p=\ln (p(1-p))\le \ln \frac{1}{4}=-2\ln 2,$$

since $p(1-p)\le \frac{1}{4}$ for $p\in (0,1)$. The other three contributions are negative, so $\ell (\mathbf{\beta })\le -2\ln 2$ for every $\mathbf{\beta }$. The likelihood can no longer be driven towards $0$ by any choice of coefficients: the conflicting pair at $x=2$ costs at least $2\ln 2$ whatever the fit does elsewhere. $■$

(d) What the output looks like. Each Newton iteration moves further out along a separating direction, so the reported coefficients grow with the number of iterations — a slope in the tens or twenties, an intercept of the opposite sign and comparable size — and the iteration halts only when the change in $\ell$ falls below the tolerance, since $\ell$ is flattening towards $0$. The fitted probabilities are numerically $0$ and $1$, and the weights $p_i(1-p_i)$ are numerically $0$, so ${\mathbf{X}}^{\top }\mathbf{WX}$ is nearly singular and its inverse — the standard errors — is enormous. Software may add a warning about fitted probabilities of $0$ or $1$.

More iterations cannot help because there is nothing to converge to: (b) shows the maximum does not exist, so any finite stopping point is arbitrary. The remedies are to change the problem — penalise the likelihood (Firth's correction or a ridge penalty, which make the penalised objective attain a finite maximum), drop or merge the offending predictor, or collect data that overlaps as in (c).

(a) The log link makes coefficients multiplicative. In the model $\log E(Y)=\eta$ the expected count is $e^{\eta }$, and increasing the checkout count by one replaces $\eta$ by $\eta +\ln 2$, so it multiplies the expected count by $e^{\ln 2}=2$. The correct sentence is “each additional checkout doubles the expected number of complaints”. The additive change is not $\ln 2$ but $E(Y)$ itself: from an expected $3$ complaints the extra checkout adds $3$, from $10$ it adds $10$. This is the same misreading as treating a logistic coefficient as a change in probability — the coefficient lives on the scale of the link, and only there is its effect constant.

(b) Unbiased coefficients do not rescue the inference. A Poisson response has variance equal to its mean, and the standard errors are computed on that assumption. If the true variance is about $6$ times the mean, the actual sampling variance of $\hat{\mathbf{\beta }}$ is about $6$ times what the model claims, so every reported standard error is too small by a factor of about $\sqrt{6}$ and every $z$-statistic too large by the same factor. The corrected statistic is

$$z=\frac{3}{\sqrt{6}}=\frac{\sqrt{6}}{2}\approx 1.22,$$

well short of $2$: the checkout effect is not significant at the $5\%$ level once the dispersion is honoured. This is the count-data twin of heteroscedasticity in the linear model — estimates fine, standard errors wrong. The remedy is a negative binomial regression, which estimates a dispersion parameter, or at least a quasi-Poisson or robust standard error that scales the Poisson ones by $\sqrt{\hat{\varphi }}$.

(c) The link is right and the distribution is wrong. The Poisson distribution lives on the counts $0,1,2,\dots$; time on a page is continuous, and a model for it that assigns probability only to whole minutes is not a model of the data. The mistake shows up concretely through the variance–mean relation, which is not invariant to the unit of measurement: a page with mean $2$ minutes and Poisson-consistent variance $2$ (min${}^2$) has, in seconds, mean $120$ and variance $2\times 60^2=7200$, not $120$. A model whose fit depends on whether you recorded seconds or minutes is measuring the units, not the behaviour.

The positive-predictions argument is correct as far as it goes — a log link does keep $E(Y)>0$ — but link and distribution are separate choices in a generalised linear model and the second was made badly. The right choices are ordinary regression on $\ln (\text{time})$, or a gamma GLM with a log link, whose variance is proportional to the square of the mean: mean $2$, variance $4\varphi$ in minutes becomes mean $120$, variance $14400\varphi =\varphi \cdot 120^2$ in seconds — the same model in both units, as a model for a continuous quantity must be.