Print this page and work through it on paper. Revealed answers are never printed, so you can check one question and still print a clean sheet.

1. 1.Computation · 6 min

   A laboratory's assay is supposed to read a reference sample at $50$ units. The measurement error is known to have standard deviation $\sigma =6$ units. A technician takes $n=36$ readings and obtains $\bar{x}=52$. Use $z_{0.05}=1.645$, $z_{0.025}=1.96$ and $z_{0.005}=2.576$.

   1. (a)Test $H_0:\mu =50$ against $H_1:\mu >50$ at $\alpha =0.05$. State the rejection region before computing the statistic.
   2. (b)Now test $H_0:\mu =50$ against the two-sided alternative, first at $\alpha =0.05$ and then at $\alpha =0.01$.
   3. (c)Write the $1\%$ two-sided verdict in one correct sentence, and explain why the sentence “at the $1\%$ level we accept that $\mu =50$” is not licensed by the test.
   Show the solution to question 1

   (a) Set up, then compute. The hypotheses are $H_0:\mu =50$ and $H_1:\mu >50$. By the one-sample $z$-test the statistic $Z=(\bar{X}-{\mu }_0)/(\sigma /\sqrt{n})$ is standard normal under $H_0$, and for an upward alternative the rejection region is $Z>z_{0.05}=1.645$.

   The standard error is $\sigma /\sqrt{n}=6/\sqrt{36}=1$, so

   $$z=\frac{52-50}{1}=2.$$

   Since $2>1.645$, reject $H_0$ at the $5\%$ level: the readings are consistent with the assay reading high.

   (b) Two-sided. The region is now $|Z|>z_{\alpha /2}$, and the statistic is unchanged, $z=2$.

   At $\alpha =0.05$: $|2|>z_{0.025}=1.96$, so reject.

   At $\alpha =0.01$: $|2|<z_{0.005}=2.576$, so fail to reject.

   The same data yield different verdicts because the standard of doubt is different, and that standard is fixed before the data are seen — it is not to be adjusted afterwards to reach a preferred verdict.

   (c) A correct sentence: “At the $1\%$ level the data do not provide sufficient evidence that the assay's mean reading differs from $50$.”

   The test controls one error only: the probability of rejecting a true null. It says nothing about how often it fails to reject a false one. Failing to reject can happen because $H_0$ is true or because the test lacks the power to see the departure — and here the second explanation is live, since $z=2$ is fairly unusual under $H_0$ (the $5\%$ test rejected). Absence of evidence is not evidence of absence.

   If one wanted to argue that $\mu$ is close to $50$, the tool is a confidence interval: the $99\%$ interval is $52\pm 2.576(1)=(49.42,54.58)$. It contains $50$, but it also contains $54$, which is hardly a demonstration that $\mu =50$.

2. 2.Computation · 6 min

   A supplier claims that at most $20\%$ of its components are defective. An inspector tests $400$ components and finds $100$ defective. Use $z_{0.025}=1.96$, $z_{0.01}=2.326$ and $z_{0.005}=2.576$.

   1. (a)Check that the normal approximation is usable, then test $H_0:p=0.2$ against $H_1:p\ne 0.2$ at $\alpha =0.05$.
   2. (b)At $\alpha =0.01$, give the verdict of the one-sided test against $H_1:p>0.2$ and of the two-sided test. They differ. Under what circumstances is the one-sided verdict the legitimate one to report?
   3. (c)Explain why the standard error in a test of a proportion uses $p_0$ and not $\hat{p}$. Then recompute the statistic with $\hat{p}$ in the denominator, give the exact value, and say what it would have done to the one-sided $1\%$ verdict.
   Show the solution to question 2

   (a) Under $H_0$, $n{p}_0=400(0.2)=80$ and $n(1-p_0)=320$, both far above $5$, so the normal approximation to the sample proportion applies.

   The observed proportion is $\hat{p}=100/400=0.25$. The standard error is computed under the null:

   $$S{E}_0=\sqrt{\frac{p_0(1-p_0)}{n}}=\sqrt{\frac{0.2\times 0.8}{400}}=\sqrt{\frac{0.16}{400}}=\frac{0.4}{20}=\mathrm{0.02.}$$

   So $z=(0.25-0.20)/0.02=2.5$. The two-sided rejection region is $|Z|>1.96$, and $2.5>1.96$, so reject $H_0$ at the $5\%$ level.

   (b) One-sided against $p>0.2$: the region is $Z>z_{0.01}=2.326$, and $2.5>2.326$, so reject. Two-sided: the region is $|Z|>z_{0.005}=2.576$, and $2.5<2.576$, so fail to reject.

   The one-sided verdict is legitimate only if the direction was fixed before the data were seen — for instance because the inspector's brief was specifically to detect a rate above the claimed one, and a rate below $20\%$ would never have been reported as a violation however small. If the direction was chosen because $\hat{p}$ happened to come out above $0.2$, the procedure rejects whenever $|Z|>2.326$, an event of probability $0.02$ under $H_0$: it is a $2\%$ test being reported as a $1\%$ one.

   (c) The null hypothesis $p=0.2$ specifies the distribution of the count completely — Binomial$(400,0.2)$ — and in particular its variance $n{p}_0(1-p_0)$. A test asks how surprising the data would be if $H_0$ held, so the null variance is the one to use. (An interval has no null value available, which is why it falls back on $\hat{p}$.)

   With $\hat{p}$ in the denominator, $\hat{p}(1-\hat{p})=0.25\times 0.75=0.1875$, so

   $$z=\frac{0.05}{\sqrt{0.1875/400}}=\frac{0.05\times 20}{\sqrt{0.1875}}=\frac{1}{\sqrt{0.1875}}=\sqrt{\frac{16}{3}}=\frac{4}{\sqrt{3}}\approx \mathrm{2.31.}$$

   It is smaller than $2.5$ because $0.1875>0.16$: the function $p(1-p)$ is largest at $p=\frac{1}{2}$, and $0.25$ is nearer to $\frac{1}{2}$ than $0.2$ is. Since $2.31<2.326$, the one-sided $1\%$ test would now fail to reject — the choice of denominator is not a technicality here, it changes the verdict.

3. 3.Computation · 8 min

   A quality engineer adopts the rule: reject $H_0:\mu =100$ in favour of $H_1:\mu >100$ whenever $\bar{X}>105$, using $n=16$ observations from a population with $\sigma =20$. Use $\Phi (1)=0.8413$, $\Phi (2)=0.9772$ and $z_{0.05}=1.645$.

   1. (a)Find the significance level $\alpha$ of this rule.
   2. (b)Find the Type II error probability $\beta$ and the power when the true mean is $110$, and again when it is $115$. In each case say which mean you standardise with, and why.
   3. (c)Keeping the rule “reject when $\bar{X}>105$” fixed, find the smallest $n$ for which $\alpha \le 0.05$. Show that at that $n$ the Type II error at $\mu =110$ is also smaller than in (b), and reconcile this with the statement that $\alpha$ and $\beta$ trade against each other.
   Show the solution to question 3

   (a) The standard error is $\sigma /\sqrt{n}=20/\sqrt{16}=5$. The significance level is the rejection probability computed under $H_0$, so standardise with ${\mu }_0=100$:

   $$\alpha =P(\bar{X}>105\mid \mu =100)=P\left(Z>\frac{105-100}{5}\right)=P(Z>1)=1-\Phi (1)=\mathrm{0.1587.}$$

   This is a lax rule: it convicts a correctly calibrated process about one time in six.

   (b) A Type II error is failing to reject when the alternative holds, so the probability is computed under the true mean, not under ${\mu }_0$: it is $\bar{X}$ that is normal with mean ${\mu }_1$, and it is $(\bar{X}-{\mu }_1)/5$ that is standard normal.

   At ${\mu }_1=110$:

   $$\beta =P(\bar{X}\le 105\mid \mu =110)=P\left(Z\le \frac{105-110}{5}\right)=P(Z\le -1)=1-\Phi (1)=0.1587,$$

   so the power is $\Phi (1)=0.8413$. (That $\alpha =\beta$ here is not a coincidence: the cutoff $105$ sits exactly halfway between $100$ and $110$, one standard error from each.)

   At ${\mu }_1=115$: $\beta =P(Z\le -2)=1-\Phi (2)=0.0228$, so the power is $0.9772$. Power rises with the distance of the alternative from the null, as it must.

   (c) With the cutoff fixed and $n$ free, $\sigma /\sqrt{n}=20/\sqrt{n}$ and

   $$\alpha (n)=P\left(Z>\frac{5}{20/\sqrt{n}}\right)=1-\Phi \left(\frac{\sqrt{n}}{4}\right).$$

   We need $\sqrt{n}/4\ge z_{0.05}=1.645$, i.e. $\sqrt{n}\ge 6.58$, i.e. $n\ge 6.58^2=43.30$. So the smallest $n$ is $44$.

   At $n=44$ the Type II error at $\mu =110$ is

   $$\beta =P\left(Z\le \frac{105-110}{20/\sqrt{44}}\right)=P\left(Z\le -\frac{\sqrt{44}}{4}\right)=1-\Phi \left(\frac{\sqrt{44}}{4}\right),$$

   and since $\sqrt{44}/4>\sqrt{16}/4=1$, this is less than $1-\Phi (1)=0.1587$. Both errors fell.

   There is no contradiction. The trade-off is a statement about the cutoff at fixed $n$: sliding $105$ upward lowers $\alpha$ and raises $\beta$, and vice versa. Increasing $n$ shrinks the standard error, so the sampling distributions under $\mu =100$ and $\mu =110$ both tighten around their means and overlap less; the tail of each beyond $105$ shrinks at once. Sample size is the lever that relaxes the trade-off, which is exactly why it is computed before a study, not after.

4. 4.Computation · 6 min

   A process has nominal mean ${\mu }_0=20$ and known $\sigma =8$. A sample of $n=64$ gives $\bar{x}=21.5$. Use $\Phi (1.5)=0.9332$.

   1. (a)Compute the $p$-value against $H_1:\mu >20$ and the $p$-value against $H_1:\mu \ne 20$. Write each as a probability statement about the test statistic before you evaluate it.
   2. (b)Give all four verdicts: each alternative at $\alpha =0.10$ and at $\alpha =0.05$.
   3. (c)A colleague writes: “$p=0.0668$, so the probability that the mean really is $20$ is about $6.7\%$, and there is a $93.3\%$ chance the effect is real.” Say precisely what the $p$-value does assert, then explain why each half of the colleague's sentence is wrong.
   Show the solution to question 4

   (a) The standard error is $8/\sqrt{64}=1$, so $z=(21.5-20)/1=1.5$.

   Against $H_1:\mu >20$, “at least as extreme” means at least as far above $0$, so

   $$p=P(Z\ge 1.5\mid H_0)=1-\Phi (1.5)=\mathrm{0.0668.}$$

   Against $H_1:\mu \ne 20$, deviations of either sign count, so by symmetry of the normal

   $$p=P(|Z|\ge 1.5\mid H_0)=2(1-\Phi (1.5))=\mathrm{0.1336.}$$

   (b) The rule is: reject exactly when $p\le \alpha$.

   One-sided: $0.0668\le 0.10$, reject at $10\%$; $0.0668>0.05$, fail to reject at $5\%$.

   Two-sided: $0.1336>0.10$ and $0.1336>0.05$, fail to reject at both levels.

   The one-sided verdicts are only available if the upward direction was specified before the data; otherwise the two-sided pair is the honest report.

   (c) What $p=0.0668$ asserts: if the true mean were exactly $20$, a sample of $64$ would produce a mean at least $1.5$ units above $20$ about $6.7\%$ of the time. It is a probability of data, computed assuming $H_0$.

   First half. “The probability that the mean is $20$” is $P(H_0\mid \text{data})$, which conditions in the opposite direction. Converting from $P(\text{data}\mid H_0)$ to $P(H_0\mid \text{data})$ needs Bayes' theorem and a prior probability for $H_0$, and the $p$-value contains no such thing. The two numbers can differ enormously.

   Second half. $1-p$ is not the probability of anything about the hypotheses. Under a true null the $p$-value is uniform on $(0,1)$, so $1-p$ is a uniform random number — it is not a level of confidence in $H_1$. The sentence is also internally inconsistent: the two-sided $p$ is $0.1336$, and the same reasoning would then give the effect an $86.6\%$ chance of being real, so the “probability the effect is real” would depend on which alternative the colleague happened to write down.

5. 5.Sketch · 9 min

   A one-sided $z$-test of $H_0:\mu =10$ against $H_1:\mu >10$ is run at $\alpha =0.05$ with $\sigma =6$. Use $z_{0.05}=1.645$ and $z_{0.20}=0.8416$.

   A sketch with no formula beside it earns nothing here: every feature you draw must be justified by an equation or a limit.

   1. (a)For $n=9$, write the power as an explicit function of the true mean ${\mu }_1$, and find the values of ${\mu }_1$ at which the power equals $0.05$, $0.5$ and $0.8$.
   2. (b)Do the same for $n=36$.
   3. (c)Sketch both power curves on one pair of axes for $8\le {\mu }_1\le 18$. Mark the three labelled points on each curve, the horizontal line at height $\alpha$, and the behaviour for ${\mu }_1<10$. State the shape of each curve and where it is steepest, and explain in one sentence how the $n=36$ curve is obtained geometrically from the $n=9$ curve.
   Show the solution to question 5

   (a) For $n=9$ the standard error is $6/\sqrt{9}=2$, so the test rejects when $\bar{X}>10+1.645(2)=13.29$. By the theorem on the power of the one-sided $z$-test,

   $$\text{power}({\mu }_1)=1-\Phi \left(1.645-\frac{{\mu }_1-10}{2}\right).$$

   Power $0.05$: we need $\Phi (\cdot )=0.95$, so the argument is $1.645$, so ${\mu }_1=10$. The power at the null is the significance level, always.

   Power $0.5$: the argument must be $0$, so $({\mu }_1-10)/2=1.645$, giving ${\mu }_1=13.29$ — the critical value itself. A true mean sitting exactly on the cutoff is detected half the time.

   Power $0.8$: we need $\Phi (a)=0.2$, so $a=-z_{0.20}=-0.8416$. Then $({\mu }_1-10)/2=1.645+0.8416=2.4866$, giving ${\mu }_1=10+4.973=14.97$.

   (b) For $n=36$ the standard error is $1$, the cutoff is $11.645$, and

   $$\text{power}({\mu }_1)=1-\Phi (1.645-({\mu }_1-10)).$$

   The three points are ${\mu }_1=10$ (power $0.05$), ${\mu }_1=11.645$ (power $0.5$) and ${\mu }_1=10+2.4866=12.49$ (power $0.8$).

   (c) Rewrite each curve using $1-\Phi (a)=\Phi (-a)$:

   $$\text{power}({\mu }_1)=\Phi \left(\frac{{\mu }_1-10}{SE}-1.645\right)=\Phi \left(\frac{{\mu }_1-c}{SE}\right),c=10+1.645SE.$$

   So each power curve is a normal distribution function in ${\mu }_1$, centred at the critical value $c$ with scale $SE$: an increasing S-shape, rising from $0$ on the far left to $1$ on the far right, steepest at its centre where the power is $0.5$ (at ${\mu }_1=13.29$ for $n=9$ and $11.645$ for $n=36$), and symmetric about that point.

   The sketch: horizontal axis ${\mu }_1$ from $8$ to $18$, vertical axis from $0$ to $1$, a dashed horizontal line at $0.05$. Both curves pass through the point $(10,0.05)$ where they cross the dashed line, since power at the null equals $\alpha$ whatever $n$ is. To the left of $10$ both curves lie below the dashed line and flatten toward $0$: at ${\mu }_1=8$ the $n=9$ curve is $\Phi (-2.645)$, essentially zero, and the $n=36$ curve is $\Phi (-3.645)$, smaller still — a true mean below the null is almost never “detected” as being above it. To the right, the $n=9$ curve passes through $(13.29,0.5)$ and $(14.97,0.8)$ and is close to $1$ by ${\mu }_1=18$ (there it is $\Phi (2.355)\approx 0.99$); the $n=36$ curve passes through $(11.645,0.5)$ and $(12.49,0.8)$ and is visually $1$ from about ${\mu }_1=14$ on.

   Geometry: since $({\mu }_1-10)/1=2\cdot ({\mu }_1-10)/2$, the $n=36$ curve at ${\mu }_1$ equals the $n=9$ curve at $10+2({\mu }_1-10)$. The $n=36$ curve is the $n=9$ curve compressed horizontally by a factor of $2$ toward the pivot ${\mu }_1=10$ — quadrupling the sample halves the effect size that can be detected at any given power, which is the $\sqrt{n}$ law.

6. 6.Computation · 9 min

   A single observation $X$ is drawn from the density $f(x;\theta )=\theta x^{\theta -1}$ on $0<x<1$, so that $\theta =1$ is the uniform distribution and the distribution function is $F(x;\theta )=x^{\theta }$. Throughout, $\alpha =0.10$.

   1. (a)Construct the most powerful level-$0.10$ test of $H_0:\theta =1$ against $H_1:\theta =2$. State the likelihood ratio, the direction in which it is monotone, the rejection region in terms of $X$, and the power.
   2. (b)A colleague proposes instead: reject when $X<0.1$. Verify that this test also has level $0.10$, compute its power against $\theta =2$, and explain what the comparison with (a) illustrates. Comment on the fact that its power is below its level.
   3. (c)Now construct the most powerful level-$0.10$ test of $H_0:\theta =1$ against $H_1:\theta =\frac{1}{2}$, and give its power exactly. Then say whether a uniformly most powerful test exists against $H_1:\theta \ne 1$, and why.
   Show the solution to question 6

   (a) The likelihood ratio of $H_1$ to $H_0$ is

   $$\Lambda (x)=\frac{f(x;2)}{f(x;1)}=\frac{2x}{1}=2x,$$

   strictly increasing in $x$. So $\{\Lambda >k\}$ is the event $\{X>c\}$ for a corresponding constant $c$, and by the Neyman–Pearson lemma the most powerful test rejects for large $X$.

   Choose $c$ for level $0.10$ under $H_0$, where $X$ is uniform: $P(X>c)=1-c=0.10$, so $c=0.9$.

   Power: under $\theta =2$, $F(x)=x^2$, so

   $$P(X>0.9\mid \theta =2)=1-0.9^2=1-0.81=\mathrm{0.19.}$$

   By the lemma, no level-$0.10$ test has power above $0.19$ against $\theta =2$.

   (b) Level: under $H_0$, $P(X<0.1)=0.1$ ✓. Power: under $\theta =2$, $P(X<0.1)=0.1^2=0.01$.

   So a test of the same level has power $0.01$ against $0.19$: the two tests spend the same Type I budget, and the likelihood-ratio test buys nineteen times as much power with it. This is the content of the lemma — controlling $\alpha$ is a constraint every admissible test satisfies, not a criterion for choosing among them, and the likelihood ratio tells you where to place the rejection region.

   The power $0.01$ is below the level $0.10$: this test is more likely to reject when $H_0$ is true than when $H_1$ is true, so its rejections are evidence in favour of the null. The reason is visible in the densities: $\theta =2$ has density $2x$, which puts very little mass near $0$, so a small observation is evidence for the uniform and against $\theta =2$. The colleague has the direction backwards.

   (c) Now

   $$\Lambda (x)=\frac{f(x;\frac{1}{2})}{f(x;1)}=\frac{\frac{1}{2}{x}^{-1/2}}{1}=\frac{1}{2\sqrt{x}},$$

   strictly decreasing in $x$. So $\{\Lambda >k\}$ is $\{X<c\}$: the most powerful test rejects for small $X$. Under $H_0$, $P(X<c)=c=0.10$, so $c=0.1$.

   Power: under $\theta =\frac{1}{2}$, $F(x)=x^{1/2}$, so

   $$P(X<0.1\mid \theta =\frac{1}{2})=\sqrt{0.1}=\frac{1}{\sqrt{10}}\approx \mathrm{0.316.}$$

   (The density $\frac{1}{2\sqrt{x}}$ is unbounded near $0$, which is why small observations are the evidence here.)

   No uniformly most powerful test exists against $\theta \ne 1$. The most powerful test against any ${\theta }_1>1$ rejects for large $X$ — the ratio ${\theta }_1{x}^{{\theta }_1-1}$ is increasing whenever ${\theta }_1>1$ — while the most powerful test against any ${\theta }_1<1$ rejects for small $X$. A single rejection region cannot be the upper tail and the lower tail at once, so no one test is best against both sides simultaneously.

7. 7.Critique · 8 min

   A student pre-registers a two-sided $z$-test of $H_0:\mu ={\mu }_0$ at $\alpha =0.05$, collects the data, and obtains $z=1.8$. Using $\Phi (1.8)=0.9641$, the write-up reads:

   Step 1. The two-sided $p$-value is $2(1-\Phi (1.8))=2(0.0359)=0.0718$, which exceeds $0.05$.

   Step 2. However, $\bar{x}$ came out above ${\mu }_0$, so the effect is clearly in the positive direction and the appropriate test is one-sided. The one-sided $p$-value is $0.0359<0.05$, so we reject $H_0$.

   Step 3. There is therefore only a $3.6\%$ chance that $H_0$ is true. The effect is real with probability $96.4\%$, and a replication of the study would succeed with the same probability.

   Step 1 is correct. Nothing after it is.

   1. (a)Identify every error in Steps 2 and 3. For each, name the definition or theorem it violates and say exactly what the quantity in question actually is.
   2. (b)Compute the true Type I error rate of the procedure used in Step 2 — “test one-sided at $5\%$ in whichever direction the data point” — and then give the correct verdict and a correct one-sentence report of the result.
   Show the solution to question 7

   (a) There are four distinct errors.

   Step 2: the direction was chosen from the data. The one-sided test's guarantee — level exactly $\alpha$ — is a statement about a rejection region fixed before the data are seen. The sentence “the effect is clearly in the positive direction” assumes the conclusion: under $H_0$, $\bar{x}$ lands above ${\mu }_0$ half the time, so the sign of $\bar{x}-{\mu }_0$ is not evidence of a direction. Picking the tail after looking at the sign is the error the chapter's caution on one-sided tests is about; its true level is computed in (b).

   Step 3, first clause: “$3.6\%$ chance that $H_0$ is true.” By definition, $p=P(T\text{ at least as extreme as }{t}_{\text{obs}}\mid H_0)$: a probability of data, computed assuming the null. The clause describes $P(H_0\mid \text{data})$, which conditions the other way round and requires a prior for $H_0$ via Bayes' theorem. The $p$-value contains no prior and cannot be converted without one.

   Step 3, second clause: “real with probability $96.4\%$.” $1-p$ is not $P(H_1\mid \text{data})$, for the same reason, and it is not a probability of anything about the hypotheses. Under a true null, $p$ is uniform on $(0,1)$ by the theorem that the $p$-value is uniform under the null, so $1-p$ is a uniform random number.

   Step 3, third clause: replication. The probability that a replication reaches significance is the power of the replication, which depends on the true effect size and the replication's sample size, neither of which $p$ encodes. If $H_0$ is in fact true, a replication “succeeds” with probability $\alpha =0.05$ — that is what uniformity of $p$ under the null means — and if $H_0$ is false the probability could be anything from near $0$ to near $1$.

   (b) The procedure rejects whenever the one-sided $p$ in the observed direction is below $0.05$, i.e. whenever $Z>z_{0.05}$ or $Z<-z_{0.05}$, i.e. whenever $|Z|>1.645$. Under $H_0$,

   $$P(|Z|>1.645)=P(Z>1.645)+P(Z<-1.645)=0.05+0.05=\mathrm{0.10.}$$

   So the “$5\%$ test” of Step 2 is a $10\%$ test: choosing the direction after the fact doubles the Type I error rate while reporting the nominal one. (Equivalently, the honest $p$-value for this procedure is the two-sided one, $0.0718$.)

   Correct verdict: the pre-registered test was two-sided, $p=0.0718>0.05$, so fail to reject $H_0$ at the $5\%$ level. A correct report: “$z=1.80$, two-sided $p=0.072$; the data do not provide sufficient evidence at the $5\%$ level that $\mu$ differs from ${\mu }_0$.” Failing to reject is not a finding that $\mu ={\mu }_0$, and a confidence interval for $\mu$ would say more than the verdict does — it would show the size of the estimated effect and how imprecisely it is pinned down.

8. 8.Computation · 10 min

   Nine subjects have a reading taken before and after a treatment.

   $$\begin{array}{lccccccccc}\text{subject}& 1& 2& 3& 4& 5& 6& 7& 8& 9\\ \text{before}& 40& 52& 38& 61& 45& 55& 47& 33& 58\\ \text{after}& 40& 53& 39& 64& 48& 59& 51& 38& 64\end{array}$$

   Use $t_{0.025,8}=2.306$ and $t_{0.025,16}=2.120$. Assume the differences are approximately normal.

   1. (a)Form the differences $D_i={\text{after}}_i-{\text{before}}_i$, compute $\bar{d}$ and $s_d$ exactly, and test $H_0:{\mu }_D=0$ against $H_1:{\mu }_D\ne 0$ at $\alpha =0.05$.
   2. (b)Write down the $95\%$ confidence interval for ${\mu }_D$ and show, by rearranging the inequality, that it gives the same verdict as (a) — and what it says that the verdict does not.
   3. (c)A colleague ignores the pairing and treats the two rows as independent samples of size $9$. Each row has standard deviation close to $10$ (the exact values are $9.6$ and $10.3$); take $s_p=10$. Compute the colleague's statistic exactly, give its verdict, and explain which analysis is valid and why the two disagree so sharply.
   Show the solution to question 8

   (a) Differences. After minus before, subject by subject: $0,1,1,3,3,4,4,5,6$. Their sum is $27$, so $\bar{d}=27/9=3$.

   Deviations from $3$: $-3,-2,-2,0,0,1,1,2,3$; their squares $9,4,4,0,0,1,1,4,9$ sum to $32$. Hence

   $$s_d^2=\frac{32}{9-1}=4,s_d=2.$$

   The test. By the paired $t$-test — the one-sample $t$-test applied to the differences — the statistic $T=\bar{D}/(S_d/\sqrt{n})$ is $t_8$ under $H_0:{\mu }_D=0$. The standard error is $2/\sqrt{9}=2/3$, so

   $$t=\frac{3}{2/3}=4.5\text{on }8\text{ degrees of freedom.}$$

   The two-sided region is $|T|>2.306$, and $4.5>2.306$, so reject $H_0$ at the $5\%$ level: the treatment changes the reading.

   (b) The $95\%$ interval is

   $$\bar{d}\pm t_{0.025,8}\frac{s_d}{\sqrt{n}}=3\pm 2.306\cdot \frac{2}{3}=3\pm 1.537=(1.46,4.54).$$

   Duality: the test rejects exactly when $|\bar{d}-0|/(s_d/\sqrt{n})>t_{0.025,8}$, and multiplying through by the positive $s_d/\sqrt{n}$ this is $|\bar{d}-0|>1.537$, i.e. $0$ lies outside $\bar{d}\pm 1.537$. Zero is indeed outside $(1.46,4.54)$, so the interval rejects too — the same inequality read two ways.

   What it adds: the test says “not zero”; the interval says the mean improvement is somewhere between about $1.5$ and $4.5$ units, which is the effect size and its precision, neither of which $t=4.5$ reports.

   (c) The difference of the row means is the mean of the differences, so the numerator is again $3$. With $s_p=10$ and $n_1=n_2=9$ the pooled standard error is

   $$s_p\sqrt{\frac{1}{9}+\frac{1}{9}}=10\sqrt{\frac{2}{9}}=\frac{10\sqrt{2}}{3}\approx 4.71,$$

   so the colleague's statistic is

   $$t=\frac{3}{10\sqrt{2}/3}=\frac{9}{10\sqrt{2}}=\frac{9\sqrt{2}}{20}\approx 0.64\text{on }9+9-2=16\text{ degrees of freedom.}$$

   Since $0.64<2.120$, the colleague fails to reject — nowhere near significance.

   The paired analysis is the valid one. The two-sample $t$-test assumes the two samples are independent, and they are not: they are the same nine people measured twice. The disagreement is not a matter of taste but of a violated hypothesis.

   Why so sharp: the readings vary a great deal between subjects (from $33$ to $61$ before treatment — a spread of about $10$), but that variation is common to both of a subject's readings and cancels completely in the difference, whose spread is only $s_d=2$. The unpaired analysis leaves the between-subject noise in the denominator, inflating the standard error by a factor of $\frac{10\sqrt{2}/3}{2/3}=5\sqrt{2}\approx 7$, and a real effect of $3$ units disappears into it.

9. 9.Computation · 9 min

   Use ${\chi }_{0.05,3}^2=7.815$, ${\chi }_{0.05,1}^2=3.841$ and ${\chi }_{0.01,1}^2=6.635$.

   1. (a)A genetic model predicts that four phenotypes occur in the ratio $9:3:3:1$. Among $160$ plants the observed counts are $100,25,27,8$. Test the model at $\alpha =0.05$, giving $X^2$ as an exact fraction, and state the conclusion in the form the test licenses.
   2. (b)Fifty patients receive treatment A and fifty receive treatment B. Under A, $36$ recover and $14$ do not; under B, $24$ recover and $26$ do not. Test whether recovery is independent of treatment at $\alpha =0.05$ and at $\alpha =0.01$.
   3. (c)Explain why the degrees of freedom are $3$ in (a) but $1$ in (b), and why all four deviations $O_{ij}-E_{ij}$ in (b) have the same magnitude. Then state the condition under which the ${\chi }^2$ approximation is trusted, and decide whether the test in (a) would still be usable if the same ratios had been observed in a sample of $40$ plants.
   Show the solution to question 9

   (a) Under $H_0$ the category probabilities are $\frac{9}{16},\frac{3}{16},\frac{3}{16},\frac{1}{16}$, so the expected counts are $E_i=160p_i$:

   $$E=90,30,30,10,$$

   all at least $5$. The deviations $O_i-E_i$ are $10,-5,-3,-2$ (they sum to $0$, as they must). By the chi-square goodness-of-fit theorem,

   $$X^2=\frac{100}{90}+\frac{25}{30}+\frac{9}{30}+\frac{4}{10}=\frac{10}{9}+\frac{5}{6}+\frac{3}{10}+\frac{2}{5}=\frac{100+75+27+36}{90}=\frac{238}{90}=\frac{119}{45}\approx \mathrm{2.64.}$$

   No parameter was estimated, so $df=k-1-m=4-1-0=3$. Since $2.64<7.815$, fail to reject: the counts are consistent with $9:3:3:1$. What is licensed is “not shown to depart from the model”, not “the model is confirmed”; and the scale is reassuring, since the mean of a ${\chi }_3^2$ is $3$ and $2.64$ is entirely typical of what the model itself would produce.

   (b) The row totals are $50$ and $50$, the column totals $60$ (recovered) and $40$ (not), and $n=100$. By the proposition on the test for independence, $E_{ij}=(\text{row total})(\text{column total})/n$:

   $$E_{11}=\frac{50\cdot 60}{100}=30,E_{12}=20,E_{21}=30,E_{22}=20,$$

   all at least $5$. The deviations are $36-30=6$, $14-20=-6$, $24-30=-6$, $26-20=6$, so

   $$X^2=\frac{36}{30}+\frac{36}{20}+\frac{36}{30}+\frac{36}{20}=1.2+1.8+1.2+1.8=6.$$

   With $df=(2-1)(2-1)=1$: $6>3.841$, so reject independence at $5\%$; $6<6.635$, so fail to reject at $1\%$. The recovery rates are $72\%$ under A and $48\%$ under B; the evidence that they differ is moderate, not overwhelming.

   (c) Degrees of freedom. Each counts the freely varying deviations. In (a) there are $4$ categories and one constraint (the counts sum to $160$), and the null specifies every probability outright, so nothing is estimated: $4-1=3$. In (b) there are $4$ cells and the same one constraint, but the null “independence” does not specify the cell probabilities — the two marginal proportions (row and column) are estimated from the data, costing one degree each: $4-1-2=1$, which is $(r-1)(c-1)$.

   Equal magnitudes. With the margins fixed, the whole table is determined by one cell: $O_{12}=50-O_{11}$, $O_{21}=60-O_{11}$, $O_{22}=O_{11}-10$. Since the expected counts satisfy the same margins, every deviation is $\pm (O_{11}-E_{11})=\pm 6$. One free cell is another way of seeing $df=1$.

   The condition. The ${\chi }^2$ distribution is an approximation, trusted when every expected count is at least about $5$; the condition is on expected counts, not observed ones. With $n=40$ the expected counts under $9:3:3:1$ would be $22.5,7.5,7.5,2.5$, and the last is below $5$: the statistic is discrete and skewed where ${\chi }_3^2$ is continuous, and the test would reject too often. The remedy is an exact multinomial test or more data — not pooling the rare category into another, which changes the hypothesis being tested.

10. 10.Proof · 12 min

    Let $X_1,\dots ,X_n$ be i.i.d. Poisson$(\lambda )$, so $P(X_i=x)={\lambda }^x{e}^{-\lambda }/x!$. Use $e^{-1}\approx 0.3679$ and $e^{-2}\approx 0.1353$ in part (c) only.

    1. (a)For ${\lambda }_1>{\lambda }_0$, write the likelihood ratio $\Lambda$ of $H_1:\lambda ={\lambda }_1$ to $H_0:\lambda ={\lambda }_0$ in terms of $S={\sum }_i{X}_i$, and prove that the Neyman–Pearson test rejects for large $S$.
    2. (b)Prove that the test in (a) is uniformly most powerful against the composite alternative $\lambda >{\lambda }_0$, and explain why no uniformly most powerful test exists against $\lambda \ne {\lambda }_0$.
    3. (c)Take $n=1$ and ${\lambda }_0=1$. Show that the rule “reject when $X\ge 3$” has level $1-\frac{5}{2}{e}^{-1}$ and the rule “reject when $X\ge 4$” has level $1-\frac{8}{3}{e}^{-1}$. Deduce that no rule of the form “reject when $X\ge c$” has level exactly $0.05$, identify the one whose level is largest without exceeding $0.05$, and find its power against $\lambda =2$ as an exact expression.
    Show the solution to question 10

    (a) The likelihood is

    $$L(\lambda )=\prod _{i=1}^n\frac{{\lambda }^{x_i}{e}^{-\lambda }}{x_i!}=\frac{{\lambda }^s{e}^{-n\lambda }}{{\prod }_i{x}_i!},s=\sum _i{x}_i.$$

    The factorials are the same under both hypotheses and cancel in the ratio:

    $$\Lambda =\frac{L({\lambda }_1)}{L({\lambda }_0)}={\left(\frac{{\lambda }_1}{{\lambda }_0}\right)}^s{e}^{-n({\lambda }_1-{\lambda }_0)},\log \Lambda =s\log \frac{{\lambda }_1}{{\lambda }_0}-n({\lambda }_1-{\lambda }_0).$$

    Since ${\lambda }_1>{\lambda }_0$, the coefficient $\log ({\lambda }_1/{\lambda }_0)$ is positive, so $\log \Lambda$ — and hence $\Lambda$ — is a strictly increasing function of $s$ alone. Therefore $\{\Lambda >k\}=\{S>c\}$ with $c=(\log k+n({\lambda }_1-{\lambda }_0))/\log ({\lambda }_1/{\lambda }_0)$. By the Neyman–Pearson lemma the most powerful test of level $\alpha$ rejects when $S$ exceeds the constant that gives level $\alpha$. $■$

    (b) The constant $c$ is fixed by the requirement $P(S>c\mid {\lambda }_0)=\alpha$, and under $H_0$ the statistic $S$ is Poisson$(n{\lambda }_0)$. So $c$ depends on ${\lambda }_0$, $n$ and $\alpha$ only — the alternative ${\lambda }_1$ dropped out, exactly as it did for the normal mean. The same rejection region $\{S>c\}$ is therefore most powerful against every ${\lambda }_1>{\lambda }_0$ at once, which is the definition of uniformly most powerful for $H_1:\lambda >{\lambda }_0$.

    For ${\lambda }_1<{\lambda }_0$ the coefficient $\log ({\lambda }_1/{\lambda }_0)$ is negative, so $\Lambda$ is strictly decreasing in $s$ and the most powerful test rejects for small $S$. The two-sided alternative contains values on both sides; the best region against one side is an upper tail and against the other a lower tail, and no single region can be both. So no test is most powerful against every $\lambda \ne {\lambda }_0$ simultaneously. $■$

    (c) Under $\lambda =1$, $P(X=k)=e^{-1}/k!$, so

    $$P(X\le 2)=e^{-1}\left(1+1+\frac{1}{2}\right)=\frac{5}{2}{e}^{-1},P(X\ge 3)=1-\frac{5}{2}{e}^{-1}\approx 1-0.9197=0.080,$$

    $$P(X\le 3)=e^{-1}\left(\frac{5}{2}+\frac{1}{6}\right)=\frac{8}{3}{e}^{-1},P(X\ge 4)=1-\frac{8}{3}{e}^{-1}\approx 1-0.9810=\mathrm{0.019.}$$

    The level of “reject when $X\ge c$” is $P(X\ge c\mid \lambda =1)$, which is decreasing in $c$ and takes only the discrete values $1,1-e^{-1},1-2e^{-1},0.080,0.019,\dots$. The value $0.05$ is not among them: $c=3$ overshoots and $c=4$ undershoots. The rule with the largest level not exceeding $0.05$ is $c=4$, with level about $0.019$.

    Its power against $\lambda =2$: $P(X=k)=2^k{e}^{-2}/k!$, so

    $$P(X\le 3\mid \lambda =2)=e^{-2}\left(1+2+2+\frac{4}{3}\right)=\frac{19}{3}{e}^{-2},\text{power}=1-\frac{19}{3}{e}^{-2}\approx 1-0.857=\mathrm{0.143.}$$

    One observation carries little information about a rate, and the discreteness has forced the test to run at level $0.019$ rather than the nominal $0.05$, which costs power on top of that.

    A remark on the lemma's wording. “$k$ chosen so that the level is exactly $\alpha$” is not achievable with a non-randomised rule when the data are discrete. The exact-level Neyman–Pearson test here rejects when $X\ge 4$, and when $X=3$ rejects with a probability $\gamma$ chosen so that $0.019+\gamma P(X=3\mid \lambda =1)=0.05$; since $P(X=3\mid \lambda =1)=e^{-1}/6\approx 0.0613$, this gives $\gamma \approx 0.51$. In practice one reports the attainable level instead.

11. 11.Proof · 11 min

    Throughout, the test statistic $T$ has a continuous distribution under $H_0$ with distribution function $F$.

    1. (a)Suppose $F$ is symmetric about $0$, so $F(-t)=1-F(t)$, and the test is two-sided with $p=P(|T|\ge |t_{\text{obs}}||H_0)=2(1-F(|t_{\text{obs}}|))$. Prove that, when $H_0$ is true, $p$ is uniform on $(0,1)$.
    2. (b)For the one-sided $z$-test rejecting when $Z>z_{\alpha }$, with observed value $z_{\text{obs}}$ and $p=1-\Phi (z_{\text{obs}})$, prove that $p$ is the smallest significance level at which the observed data lead to rejection: $p=\inf \{\alpha \in (0,1):z_{\text{obs}}>z_{\alpha }\}$.
    3. (c)A journal publishes a result only if its $p$-value is at most $0.05$. Suppose every null hypothesis submitted is in fact true. Prove that the $p$-values appearing in the journal are uniform on $(0,0.05)$, find the fraction of published $p$-values below $0.01$ and the fraction in $(0.04,0.05]$, and say what an observed pile-up of published $p$-values just below $0.05$ would therefore indicate.
    Show the solution to question 11

    (a) Let $G(t)=P(|T|\le t\mid H_0)$ for $t\ge 0$. By symmetry,

    $$G(t)=P(-t\le T\le t)=F(t)-F(-t)=2F(t)-1,$$

    which is continuous and strictly increasing on the support of $|T|$, from $0$ to $1$. The two-sided $p$-value is $p=P(|T|\ge |t_{\text{obs}}|)=1-G(|t_{\text{obs}}|)=2(1-F(|t_{\text{obs}}|))$, matching the stated formula; as a random variable, $p=1-G(|T|)$.

    Now fix $u\in (0,1)$. Because $G$ is continuous and strictly increasing it has an inverse on $(0,1)$, and

    $$P(p\le u)=P(G(|T|)\ge 1-u)=P(|T|\ge G^{-1}(1-u))=1-G(G^{-1}(1-u))=u,$$

    where the third equality uses continuity, so that $P(|T|=G^{-1}(1-u))=0$ and the inequality may be taken strict or not. A random variable with $P(p\le u)=u$ on $(0,1)$ is uniform on $(0,1)$. $■$

    The argument is the one-sided proof applied to the statistic $|T|$: symmetry was used only to write $G$ in terms of $F$. In particular $P(p\le \alpha \mid H_0)=\alpha$, so “reject when $p\le \alpha$” is a level-$\alpha$ test, and it coincides with rejecting when $|T|>G^{-1}(1-\alpha )$, the two-sided critical value.

    (b) By definition $z_{\alpha }={\Phi }^{-1}(1-\alpha )$, which is a strictly decreasing function of $\alpha$ because ${\Phi }^{-1}$ is increasing. Then, for $\alpha \in (0,1)$,

    $$z_{\text{obs}}>z_{\alpha }⟺\Phi (z_{\text{obs}})>1-\alpha ⟺\alpha >1-\Phi (z_{\text{obs}})=p,$$

    applying the increasing function $\Phi$ in the first step and rearranging in the second. So the set of levels at which the data are rejected is exactly the interval $(p,1)$, whose infimum is $p$. $■$

    At $\alpha =p$ itself the strict rule $Z>z_{\alpha }$ does not reject — $z_{\text{obs}}=z_p$ exactly — which is why the statement is an infimum rather than a minimum; the distinction has probability zero and is invisible in practice. This is the operational meaning of a $p$-value: it is the level at which the verdict flips, so reporting $p$ tells every reader the verdict at their own $\alpha$.

    (c) Let $p$ be the $p$-value of a submitted test. Its null is true, so by the theorem that the $p$-value is uniform under the null (or by (a)), $p\sim \text{Uniform}(0,1)$. Publication is the event $\{p\le 0.05\}$, of probability $0.05$. For $0<u\le 0.05$,

    $$P(p\le u\mid p\le 0.05)=\frac{P(p\le u\text{and}p\le 0.05)}{P(p\le 0.05)}=\frac{P(p\le u)}{0.05}=\frac{u}{0.05},$$

    and for $u>0.05$ the conditional probability is $1$. This is the distribution function of Uniform$(0,0.05)$. $■$

    Hence $P(p<0.01\mid \text{published})=0.01/0.05=\frac{1}{5}$, and likewise $P(0.04<p\le 0.05\mid \text{published})=0.01/0.05=\frac{1}{5}$: each fifth of the interval receives $20\%$ of the published values.

    Interpretation. If the published nulls were all true and the filter were the only selection, the published $p$-values would be flat across $(0,0.05)$. If instead some effects were real, their $p$-values would concentrate near $0$ — under a real effect the distribution of $p$ piles up at the low end — and the published histogram would be heavier below $0.01$ than just below $0.05$. Neither honest scenario produces an excess just under the threshold. A pile-up in $(0.04,0.05]$ relative to $(0,0.01)$ is therefore the signature of results that were nudged across the line: choosing among analyses, subgroups, or stopping times until $p$ dipped below $0.05$, which the printed $p$-value does not record.

12. 12.Computation · 12 min

    Use $\ln 2\approx 0.6931$, $\ln 0.95\approx -0.0513$ and $(0.95{)}^{20}\approx 0.3585$.

    1. (a)A study runs $m$ independent tests, each at level $\alpha$, and every null is true. Show that the number $K$ of false rejections is Binomial$(m,\alpha )$, and find $E[K]$ and $P(K\ge 1)$ for $m=20$, $\alpha =0.05$. Then find the smallest $m$ for which $P(K\ge 1)\ge \frac{1}{2}$ at $\alpha =0.05$.
    2. (b)Prove that testing each hypothesis at level $\alpha /m$ keeps the probability of at least one false rejection at most $\alpha$ under any dependence between the tests. Then, for independent tests with all nulls true, write the exact family-wise error rate under this correction and prove from Bernoulli's inequality $(1-x{)}^m\ge 1-mx$ (valid for $0\le x\le 1$) that it is at most $\alpha$ — so the correction is conservative even in the independent case.
    3. (c)In a field, a fraction $\pi$ of tested hypotheses are genuinely non-null, and tests have level $\alpha$ and power $1-\beta$. Show that a rejected null is more likely true than false — that is, $P(H_1\mid \text{reject})\ge \frac{1}{2}$ — exactly when $(1-\beta )\pi \ge \alpha (1-\pi )$. With $\pi =0.05$, find the power required at $\alpha =0.05$ and at $\alpha =0.005$, and draw the conclusion.
    Show the solution to question 12

    (a) Let $A_i$ be the event that test $i$ falsely rejects, and $I_i$ its indicator. Since every null is true, $P(A_i)=\alpha$, so each $I_i$ is Bernoulli$(\alpha )$; the tests are independent, so the $I_i$ are independent; and $K={\sum }_{i=1}^m{I}_i$ is a sum of $m$ independent Bernoulli$(\alpha )$ variables, which is Binomial$(m,\alpha )$.

    For $m=20$, $\alpha =0.05$: $E[K]=m\alpha =1$ — on average one false finding per study of pure noise — and

    $$P(K\ge 1)=1-P(K=0)=1-(1-\alpha {)}^m=1-(0.95{)}^{20}\approx 1-0.3585=0.6415,$$

    which is the family-wise error rate under multiplicity.

    Smallest $m$ with $P(K\ge 1)\ge \frac{1}{2}$: we need $(0.95{)}^m\le \frac{1}{2}$. Taking logarithms (which reverses the inequality, since $\ln 0.95<0$),

    $$m\ln 0.95\le -\ln 2⟺m\ge \frac{\ln 2}{-\ln 0.95}\approx \frac{0.6931}{0.0513}\approx 13.5,$$

    so $m=14$. Check: $(0.95{)}^{10}\approx 0.5987$ and $(0.95{)}^3\approx 0.8574$, so $(0.95{)}^{13}\approx 0.513>\frac{1}{2}$ while $(0.95{)}^{14}\approx 0.488<\frac{1}{2}$ ✓. Fourteen honest tests on noise are more likely than not to yield a “finding”.

    (b) Bonferroni under arbitrary dependence. With each test at level $\alpha /m$, $P(A_i)\le \alpha /m$ for every true null. By the union bound — subadditivity of probability, which needs no independence —

    $$P\left(\bigcup _{i=1}^m{A}_i\right)\le \sum _{i=1}^{m}P(A_i)\le m\cdot \frac{\alpha }{m}=\alpha .■$$

    The independent case. By (a) with $\alpha /m$ in place of $\alpha$, the number of false rejections is Binomial$(m,\alpha /m)$ and the exact family-wise error rate is

    $$1-{\left(1-\frac{\alpha }{m}\right)}^m.$$

    Bernoulli's inequality with $x=\alpha /m\in [0,1]$ gives $(1-\alpha /m{)}^m\ge 1-m\cdot \alpha /m=1-\alpha$, hence

    $$1-{\left(1-\frac{\alpha }{m}\right)}^m\le \alpha .■$$

    (For completeness, Bernoulli's inequality by induction: it is an equality at $m=1$, and if $(1-x{)}^m\ge 1-mx$ then, multiplying by $1-x\ge 0$, $(1-x{)}^{m+1}\ge (1-mx)(1-x)=1-(m+1)x+m{x}^2\ge 1-(m+1)x$.)

    The inequality is strict for $m\ge 2$ and $x>0$, so even independent tests do not use the whole budget: at $m=20$, $\alpha =0.05$ the true rate is $1-(0.9975{)}^{20}\approx 0.049$. The shortfall is small here — of order ${\alpha }^2$ — but it grows when the tests are positively correlated, where the union bound is far from tight and the correction costs power for a guarantee that was already met.

    (c) By the post-test probability theorem,

    $$P(H_1\mid \text{reject})=\frac{(1-\beta )\pi }{(1-\beta )\pi +\alpha (1-\pi )}.$$

    This is at least $\frac{1}{2}$ exactly when the numerator is at least half the denominator, i.e. $2(1-\beta )\pi \ge (1-\beta )\pi +\alpha (1-\pi )$, i.e.

    $$(1-\beta )\pi \ge \alpha (1-\pi ),$$

    and every step reverses, since the denominator is positive. In words: the rate of true positives must at least match the rate of false positives.

    With $\pi =0.05$: the false-positive rate is $\alpha (1-\pi )=0.95\alpha$.

    At $\alpha =0.05$: $0.05(1-\beta )\ge 0.05(0.95)$, so the power must be at least $0.95$.

    At $\alpha =0.005$: $0.05(1-\beta )\ge 0.005(0.95)=0.00475$, so the power must be at least $0.095$.

    Conclusion. In a field that tests long shots — one hypothesis in twenty being real — a significant result at the conventional $5\%$ level is more likely false than true unless the study has $95\%$ power, which almost no study has; and even at perfect power the post-test probability is only $\frac{0.05}{0.05+0.0475}\approx 0.51$. Power cannot rescue such a literature, because the false positives come from the large pool of true nulls, not from the small pool of real effects. Lowering $\alpha$ tenfold shrinks that pool's contribution tenfold, and then even a modest $10\%$ power makes a rejection more likely right than wrong. The $p$-value's meaning depends on two things it does not contain: how plausible the hypothesis was before the data, and how powerful the test was.