Statistical Inference: Hypothesis Testing

(a) Set up, then compute. The hypotheses are $H_0:\mu =50$ and $H_1:\mu >50$. By the one-sample $z$-test the statistic $Z=(\bar{X}-{\mu }_0)/(\sigma /\sqrt{n})$ is standard normal under $H_0$, and for an upward alternative the rejection region is $Z>z_{0.05}=1.645$.

The standard error is $\sigma /\sqrt{n}=6/\sqrt{36}=1$, so

$$z=\frac{52-50}{1}=2.$$

Since $2>1.645$, reject $H_0$ at the $5\%$ level: the readings are consistent with the assay reading high.

(b) Two-sided. The region is now $|Z|>z_{\alpha /2}$, and the statistic is unchanged, $z=2$.

At $\alpha =0.05$: $|2|>z_{0.025}=1.96$, so reject.

At $\alpha =0.01$: $|2|<z_{0.005}=2.576$, so fail to reject.

The same data yield different verdicts because the standard of doubt is different, and that standard is fixed before the data are seen — it is not to be adjusted afterwards to reach a preferred verdict.

(c) A correct sentence: “At the $1\%$ level the data do not provide sufficient evidence that the assay's mean reading differs from $50$.”

The test controls one error only: the probability of rejecting a true null. It says nothing about how often it fails to reject a false one. Failing to reject can happen because $H_0$ is true or because the test lacks the power to see the departure — and here the second explanation is live, since $z=2$ is fairly unusual under $H_0$ (the $5\%$ test rejected). Absence of evidence is not evidence of absence.

If one wanted to argue that $\mu$ is close to $50$, the tool is a confidence interval: the $99\%$ interval is $52\pm 2.576(1)=(49.42,54.58)$. It contains $50$, but it also contains $54$, which is hardly a demonstration that $\mu =50$.

(a) Under $H_0$, $n{p}_0=400(0.2)=80$ and $n(1-p_0)=320$, both far above $5$, so the normal approximation to the sample proportion applies.

The observed proportion is $\hat{p}=100/400=0.25$. The standard error is computed under the null:

$$S{E}_0=\sqrt{\frac{p_0(1-p_0)}{n}}=\sqrt{\frac{0.2\times 0.8}{400}}=\sqrt{\frac{0.16}{400}}=\frac{0.4}{20}=\mathrm{0.02.}$$

So $z=(0.25-0.20)/0.02=2.5$. The two-sided rejection region is $|Z|>1.96$, and $2.5>1.96$, so reject $H_0$ at the $5\%$ level.

(b) One-sided against $p>0.2$: the region is $Z>z_{0.01}=2.326$, and $2.5>2.326$, so reject. Two-sided: the region is $|Z|>z_{0.005}=2.576$, and $2.5<2.576$, so fail to reject.

The one-sided verdict is legitimate only if the direction was fixed before the data were seen — for instance because the inspector's brief was specifically to detect a rate above the claimed one, and a rate below $20\%$ would never have been reported as a violation however small. If the direction was chosen because $\hat{p}$ happened to come out above $0.2$, the procedure rejects whenever $|Z|>2.326$, an event of probability $0.02$ under $H_0$: it is a $2\%$ test being reported as a $1\%$ one.

(c) The null hypothesis $p=0.2$ specifies the distribution of the count completely — Binomial$(400,0.2)$ — and in particular its variance $n{p}_0(1-p_0)$. A test asks how surprising the data would be if $H_0$ held, so the null variance is the one to use. (An interval has no null value available, which is why it falls back on $\hat{p}$.)

With $\hat{p}$ in the denominator, $\hat{p}(1-\hat{p})=0.25\times 0.75=0.1875$, so

$$z=\frac{0.05}{\sqrt{0.1875/400}}=\frac{0.05\times 20}{\sqrt{0.1875}}=\frac{1}{\sqrt{0.1875}}=\sqrt{\frac{16}{3}}=\frac{4}{\sqrt{3}}\approx \mathrm{2.31.}$$

It is smaller than $2.5$ because $0.1875>0.16$: the function $p(1-p)$ is largest at $p=\frac{1}{2}$, and $0.25$ is nearer to $\frac{1}{2}$ than $0.2$ is. Since $2.31<2.326$, the one-sided $1\%$ test would now fail to reject — the choice of denominator is not a technicality here, it changes the verdict.

(a) The standard error is $\sigma /\sqrt{n}=20/\sqrt{16}=5$. The significance level is the rejection probability computed under $H_0$, so standardise with ${\mu }_0=100$:

$$\alpha =P(\bar{X}>105\mid \mu =100)=P\left(Z>\frac{105-100}{5}\right)=P(Z>1)=1-\Phi (1)=\mathrm{0.1587.}$$

This is a lax rule: it convicts a correctly calibrated process about one time in six.

(b) A Type II error is failing to reject when the alternative holds, so the probability is computed under the true mean, not under ${\mu }_0$: it is $\bar{X}$ that is normal with mean ${\mu }_1$, and it is $(\bar{X}-{\mu }_1)/5$ that is standard normal.

At ${\mu }_1=110$:

$$\beta =P(\bar{X}\le 105\mid \mu =110)=P\left(Z\le \frac{105-110}{5}\right)=P(Z\le -1)=1-\Phi (1)=0.1587,$$

so the power is $\Phi (1)=0.8413$. (That $\alpha =\beta$ here is not a coincidence: the cutoff $105$ sits exactly halfway between $100$ and $110$, one standard error from each.)

At ${\mu }_1=115$: $\beta =P(Z\le -2)=1-\Phi (2)=0.0228$, so the power is $0.9772$. Power rises with the distance of the alternative from the null, as it must.

(c) With the cutoff fixed and $n$ free, $\sigma /\sqrt{n}=20/\sqrt{n}$ and

$$\alpha (n)=P\left(Z>\frac{5}{20/\sqrt{n}}\right)=1-\Phi \left(\frac{\sqrt{n}}{4}\right).$$

We need $\sqrt{n}/4\ge z_{0.05}=1.645$, i.e. $\sqrt{n}\ge 6.58$, i.e. $n\ge 6.58^2=43.30$. So the smallest $n$ is $44$.

At $n=44$ the Type II error at $\mu =110$ is

$$\beta =P\left(Z\le \frac{105-110}{20/\sqrt{44}}\right)=P\left(Z\le -\frac{\sqrt{44}}{4}\right)=1-\Phi \left(\frac{\sqrt{44}}{4}\right),$$

and since $\sqrt{44}/4>\sqrt{16}/4=1$, this is less than $1-\Phi (1)=0.1587$. Both errors fell.

There is no contradiction. The trade-off is a statement about the cutoff at fixed $n$: sliding $105$ upward lowers $\alpha$ and raises $\beta$, and vice versa. Increasing $n$ shrinks the standard error, so the sampling distributions under $\mu =100$ and $\mu =110$ both tighten around their means and overlap less; the tail of each beyond $105$ shrinks at once. Sample size is the lever that relaxes the trade-off, which is exactly why it is computed before a study, not after.

(a) The standard error is $8/\sqrt{64}=1$, so $z=(21.5-20)/1=1.5$.

Against $H_1:\mu >20$, “at least as extreme” means at least as far above $0$, so

$$p=P(Z\ge 1.5\mid H_0)=1-\Phi (1.5)=\mathrm{0.0668.}$$

Against $H_1:\mu \ne 20$, deviations of either sign count, so by symmetry of the normal

$$p=P(|Z|\ge 1.5\mid H_0)=2(1-\Phi (1.5))=\mathrm{0.1336.}$$

(b) The rule is: reject exactly when $p\le \alpha$.

One-sided: $0.0668\le 0.10$, reject at $10\%$; $0.0668>0.05$, fail to reject at $5\%$.

Two-sided: $0.1336>0.10$ and $0.1336>0.05$, fail to reject at both levels.

The one-sided verdicts are only available if the upward direction was specified before the data; otherwise the two-sided pair is the honest report.

(c) What $p=0.0668$ asserts: if the true mean were exactly $20$, a sample of $64$ would produce a mean at least $1.5$ units above $20$ about $6.7\%$ of the time. It is a probability of data, computed assuming $H_0$.

First half. “The probability that the mean is $20$” is $P(H_0\mid \text{data})$, which conditions in the opposite direction. Converting from $P(\text{data}\mid H_0)$ to $P(H_0\mid \text{data})$ needs Bayes' theorem and a prior probability for $H_0$, and the $p$-value contains no such thing. The two numbers can differ enormously.

Second half. $1-p$ is not the probability of anything about the hypotheses. Under a true null the $p$-value is uniform on $(0,1)$, so $1-p$ is a uniform random number — it is not a level of confidence in $H_1$. The sentence is also internally inconsistent: the two-sided $p$ is $0.1336$, and the same reasoning would then give the effect an $86.6\%$ chance of being real, so the “probability the effect is real” would depend on which alternative the colleague happened to write down.

(a) For $n=9$ the standard error is $6/\sqrt{9}=2$, so the test rejects when $\bar{X}>10+1.645(2)=13.29$. By the theorem on the power of the one-sided $z$-test,

$$\text{power}({\mu }_1)=1-\Phi \left(1.645-\frac{{\mu }_1-10}{2}\right).$$

Power $0.05$: we need $\Phi (\cdot )=0.95$, so the argument is $1.645$, so ${\mu }_1=10$. The power at the null is the significance level, always.

Power $0.5$: the argument must be $0$, so $({\mu }_1-10)/2=1.645$, giving ${\mu }_1=13.29$ — the critical value itself. A true mean sitting exactly on the cutoff is detected half the time.

Power $0.8$: we need $\Phi (a)=0.2$, so $a=-z_{0.20}=-0.8416$. Then $({\mu }_1-10)/2=1.645+0.8416=2.4866$, giving ${\mu }_1=10+4.973=14.97$.

(b) For $n=36$ the standard error is $1$, the cutoff is $11.645$, and

$$\text{power}({\mu }_1)=1-\Phi (1.645-({\mu }_1-10)).$$

The three points are ${\mu }_1=10$ (power $0.05$), ${\mu }_1=11.645$ (power $0.5$) and ${\mu }_1=10+2.4866=12.49$ (power $0.8$).

(c) Rewrite each curve using $1-\Phi (a)=\Phi (-a)$:

$$\text{power}({\mu }_1)=\Phi \left(\frac{{\mu }_1-10}{SE}-1.645\right)=\Phi \left(\frac{{\mu }_1-c}{SE}\right),c=10+1.645SE.$$

So each power curve is a normal distribution function in ${\mu }_1$, centred at the critical value $c$ with scale $SE$: an increasing S-shape, rising from $0$ on the far left to $1$ on the far right, steepest at its centre where the power is $0.5$ (at ${\mu }_1=13.29$ for $n=9$ and $11.645$ for $n=36$), and symmetric about that point.

The sketch: horizontal axis ${\mu }_1$ from $8$ to $18$, vertical axis from $0$ to $1$, a dashed horizontal line at $0.05$. Both curves pass through the point $(10,0.05)$ where they cross the dashed line, since power at the null equals $\alpha$ whatever $n$ is. To the left of $10$ both curves lie below the dashed line and flatten toward $0$: at ${\mu }_1=8$ the $n=9$ curve is $\Phi (-2.645)$, essentially zero, and the $n=36$ curve is $\Phi (-3.645)$, smaller still — a true mean below the null is almost never “detected” as being above it. To the right, the $n=9$ curve passes through $(13.29,0.5)$ and $(14.97,0.8)$ and is close to $1$ by ${\mu }_1=18$ (there it is $\Phi (2.355)\approx 0.99$); the $n=36$ curve passes through $(11.645,0.5)$ and $(12.49,0.8)$ and is visually $1$ from about ${\mu }_1=14$ on.

Geometry: since $({\mu }_1-10)/1=2\cdot ({\mu }_1-10)/2$, the $n=36$ curve at ${\mu }_1$ equals the $n=9$ curve at $10+2({\mu }_1-10)$. The $n=36$ curve is the $n=9$ curve compressed horizontally by a factor of $2$ toward the pivot ${\mu }_1=10$ — quadrupling the sample halves the effect size that can be detected at any given power, which is the $\sqrt{n}$ law.

(a) The likelihood ratio of $H_1$ to $H_0$ is

$$\Lambda (x)=\frac{f(x;2)}{f(x;1)}=\frac{2x}{1}=2x,$$

strictly increasing in $x$. So $\{\Lambda >k\}$ is the event $\{X>c\}$ for a corresponding constant $c$, and by the Neyman–Pearson lemma the most powerful test rejects for large $X$.

Choose $c$ for level $0.10$ under $H_0$, where $X$ is uniform: $P(X>c)=1-c=0.10$, so $c=0.9$.

Power: under $\theta =2$, $F(x)=x^2$, so

$$P(X>0.9\mid \theta =2)=1-0.9^2=1-0.81=\mathrm{0.19.}$$

By the lemma, no level-$0.10$ test has power above $0.19$ against $\theta =2$.

(b) Level: under $H_0$, $P(X<0.1)=0.1$ ✓. Power: under $\theta =2$, $P(X<0.1)=0.1^2=0.01$.

So a test of the same level has power $0.01$ against $0.19$: the two tests spend the same Type I budget, and the likelihood-ratio test buys nineteen times as much power with it. This is the content of the lemma — controlling $\alpha$ is a constraint every admissible test satisfies, not a criterion for choosing among them, and the likelihood ratio tells you where to place the rejection region.

The power $0.01$ is below the level $0.10$: this test is more likely to reject when $H_0$ is true than when $H_1$ is true, so its rejections are evidence in favour of the null. The reason is visible in the densities: $\theta =2$ has density $2x$, which puts very little mass near $0$, so a small observation is evidence for the uniform and against $\theta =2$. The colleague has the direction backwards.

(c) Now

$$\Lambda (x)=\frac{f(x;\frac{1}{2})}{f(x;1)}=\frac{\frac{1}{2}{x}^{-1/2}}{1}=\frac{1}{2\sqrt{x}},$$

strictly decreasing in $x$. So $\{\Lambda >k\}$ is $\{X<c\}$: the most powerful test rejects for small $X$. Under $H_0$, $P(X<c)=c=0.10$, so $c=0.1$.

Power: under $\theta =\frac{1}{2}$, $F(x)=x^{1/2}$, so

$$P(X<0.1\mid \theta =\frac{1}{2})=\sqrt{0.1}=\frac{1}{\sqrt{10}}\approx \mathrm{0.316.}$$

(The density $\frac{1}{2\sqrt{x}}$ is unbounded near $0$, which is why small observations are the evidence here.)

No uniformly most powerful test exists against $\theta \ne 1$. The most powerful test against any ${\theta }_1>1$ rejects for large $X$ — the ratio ${\theta }_1{x}^{{\theta }_1-1}$ is increasing whenever ${\theta }_1>1$ — while the most powerful test against any ${\theta }_1<1$ rejects for small $X$. A single rejection region cannot be the upper tail and the lower tail at once, so no one test is best against both sides simultaneously.

(a) There are four distinct errors.

Step 2: the direction was chosen from the data. The one-sided test's guarantee — level exactly $\alpha$ — is a statement about a rejection region fixed before the data are seen. The sentence “the effect is clearly in the positive direction” assumes the conclusion: under $H_0$, $\bar{x}$ lands above ${\mu }_0$ half the time, so the sign of $\bar{x}-{\mu }_0$ is not evidence of a direction. Picking the tail after looking at the sign is the error the chapter's caution on one-sided tests is about; its true level is computed in (b).

Step 3, first clause: “$3.6\%$ chance that $H_0$ is true.” By definition, $p=P(T\text{ at least as extreme as }{t}_{\text{obs}}\mid H_0)$: a probability of data, computed assuming the null. The clause describes $P(H_0\mid \text{data})$, which conditions the other way round and requires a prior for $H_0$ via Bayes' theorem. The $p$-value contains no prior and cannot be converted without one.

Step 3, second clause: “real with probability $96.4\%$.” $1-p$ is not $P(H_1\mid \text{data})$, for the same reason, and it is not a probability of anything about the hypotheses. Under a true null, $p$ is uniform on $(0,1)$ by the theorem that the $p$-value is uniform under the null, so $1-p$ is a uniform random number.

Step 3, third clause: replication. The probability that a replication reaches significance is the power of the replication, which depends on the true effect size and the replication's sample size, neither of which $p$ encodes. If $H_0$ is in fact true, a replication “succeeds” with probability $\alpha =0.05$ — that is what uniformity of $p$ under the null means — and if $H_0$ is false the probability could be anything from near $0$ to near $1$.

(b) The procedure rejects whenever the one-sided $p$ in the observed direction is below $0.05$, i.e. whenever $Z>z_{0.05}$ or $Z<-z_{0.05}$, i.e. whenever $|Z|>1.645$. Under $H_0$,

$$P(|Z|>1.645)=P(Z>1.645)+P(Z<-1.645)=0.05+0.05=\mathrm{0.10.}$$

So the “$5\%$ test” of Step 2 is a $10\%$ test: choosing the direction after the fact doubles the Type I error rate while reporting the nominal one. (Equivalently, the honest $p$-value for this procedure is the two-sided one, $0.0718$.)

Correct verdict: the pre-registered test was two-sided, $p=0.0718>0.05$, so fail to reject $H_0$ at the $5\%$ level. A correct report: “$z=1.80$, two-sided $p=0.072$; the data do not provide sufficient evidence at the $5\%$ level that $\mu$ differs from ${\mu }_0$.” Failing to reject is not a finding that $\mu ={\mu }_0$, and a confidence interval for $\mu$ would say more than the verdict does — it would show the size of the estimated effect and how imprecisely it is pinned down.

(a) Differences. After minus before, subject by subject: $0,1,1,3,3,4,4,5,6$. Their sum is $27$, so $\bar{d}=27/9=3$.

Deviations from $3$: $-3,-2,-2,0,0,1,1,2,3$; their squares $9,4,4,0,0,1,1,4,9$ sum to $32$. Hence

$$s_d^2=\frac{32}{9-1}=4,s_d=2.$$

The test. By the paired $t$-test — the one-sample $t$-test applied to the differences — the statistic $T=\bar{D}/(S_d/\sqrt{n})$ is $t_8$ under $H_0:{\mu }_D=0$. The standard error is $2/\sqrt{9}=2/3$, so

$$t=\frac{3}{2/3}=4.5\text{on }8\text{ degrees of freedom.}$$

The two-sided region is $|T|>2.306$, and $4.5>2.306$, so reject $H_0$ at the $5\%$ level: the treatment changes the reading.

(b) The $95\%$ interval is

$$\bar{d}\pm t_{0.025,8}\frac{s_d}{\sqrt{n}}=3\pm 2.306\cdot \frac{2}{3}=3\pm 1.537=(1.46,4.54).$$

Duality: the test rejects exactly when $|\bar{d}-0|/(s_d/\sqrt{n})>t_{0.025,8}$, and multiplying through by the positive $s_d/\sqrt{n}$ this is $|\bar{d}-0|>1.537$, i.e. $0$ lies outside $\bar{d}\pm 1.537$. Zero is indeed outside $(1.46,4.54)$, so the interval rejects too — the same inequality read two ways.

What it adds: the test says “not zero”; the interval says the mean improvement is somewhere between about $1.5$ and $4.5$ units, which is the effect size and its precision, neither of which $t=4.5$ reports.

(c) The difference of the row means is the mean of the differences, so the numerator is again $3$. With $s_p=10$ and $n_1=n_2=9$ the pooled standard error is

$$s_p\sqrt{\frac{1}{9}+\frac{1}{9}}=10\sqrt{\frac{2}{9}}=\frac{10\sqrt{2}}{3}\approx 4.71,$$

so the colleague's statistic is

$$t=\frac{3}{10\sqrt{2}/3}=\frac{9}{10\sqrt{2}}=\frac{9\sqrt{2}}{20}\approx 0.64\text{on }9+9-2=16\text{ degrees of freedom.}$$

Since $0.64<2.120$, the colleague fails to reject — nowhere near significance.

The paired analysis is the valid one. The two-sample $t$-test assumes the two samples are independent, and they are not: they are the same nine people measured twice. The disagreement is not a matter of taste but of a violated hypothesis.

Why so sharp: the readings vary a great deal between subjects (from $33$ to $61$ before treatment — a spread of about $10$), but that variation is common to both of a subject's readings and cancels completely in the difference, whose spread is only $s_d=2$. The unpaired analysis leaves the between-subject noise in the denominator, inflating the standard error by a factor of $\frac{10\sqrt{2}/3}{2/3}=5\sqrt{2}\approx 7$, and a real effect of $3$ units disappears into it.

(a) Under $H_0$ the category probabilities are $\frac{9}{16},\frac{3}{16},\frac{3}{16},\frac{1}{16}$, so the expected counts are $E_i=160p_i$:

$$E=90,30,30,10,$$

all at least $5$. The deviations $O_i-E_i$ are $10,-5,-3,-2$ (they sum to $0$, as they must). By the chi-square goodness-of-fit theorem,

$$X^2=\frac{100}{90}+\frac{25}{30}+\frac{9}{30}+\frac{4}{10}=\frac{10}{9}+\frac{5}{6}+\frac{3}{10}+\frac{2}{5}=\frac{100+75+27+36}{90}=\frac{238}{90}=\frac{119}{45}\approx \mathrm{2.64.}$$

No parameter was estimated, so $df=k-1-m=4-1-0=3$. Since $2.64<7.815$, fail to reject: the counts are consistent with $9:3:3:1$. What is licensed is “not shown to depart from the model”, not “the model is confirmed”; and the scale is reassuring, since the mean of a ${\chi }_3^2$ is $3$ and $2.64$ is entirely typical of what the model itself would produce.

(b) The row totals are $50$ and $50$, the column totals $60$ (recovered) and $40$ (not), and $n=100$. By the proposition on the test for independence, $E_{ij}=(\text{row total})(\text{column total})/n$:

$$E_{11}=\frac{50\cdot 60}{100}=30,E_{12}=20,E_{21}=30,E_{22}=20,$$

all at least $5$. The deviations are $36-30=6$, $14-20=-6$, $24-30=-6$, $26-20=6$, so

$$X^2=\frac{36}{30}+\frac{36}{20}+\frac{36}{30}+\frac{36}{20}=1.2+1.8+1.2+1.8=6.$$

With $df=(2-1)(2-1)=1$: $6>3.841$, so reject independence at $5\%$; $6<6.635$, so fail to reject at $1\%$. The recovery rates are $72\%$ under A and $48\%$ under B; the evidence that they differ is moderate, not overwhelming.

(c) Degrees of freedom. Each counts the freely varying deviations. In (a) there are $4$ categories and one constraint (the counts sum to $160$), and the null specifies every probability outright, so nothing is estimated: $4-1=3$. In (b) there are $4$ cells and the same one constraint, but the null “independence” does not specify the cell probabilities — the two marginal proportions (row and column) are estimated from the data, costing one degree each: $4-1-2=1$, which is $(r-1)(c-1)$.

Equal magnitudes. With the margins fixed, the whole table is determined by one cell: $O_{12}=50-O_{11}$, $O_{21}=60-O_{11}$, $O_{22}=O_{11}-10$. Since the expected counts satisfy the same margins, every deviation is $\pm (O_{11}-E_{11})=\pm 6$. One free cell is another way of seeing $df=1$.

The condition. The ${\chi }^2$ distribution is an approximation, trusted when every expected count is at least about $5$; the condition is on expected counts, not observed ones. With $n=40$ the expected counts under $9:3:3:1$ would be $22.5,7.5,7.5,2.5$, and the last is below $5$: the statistic is discrete and skewed where ${\chi }_3^2$ is continuous, and the test would reject too often. The remedy is an exact multinomial test or more data — not pooling the rare category into another, which changes the hypothesis being tested.

(a) The likelihood is

$$L(\lambda )=\prod _{i=1}^n\frac{{\lambda }^{x_i}{e}^{-\lambda }}{x_i!}=\frac{{\lambda }^s{e}^{-n\lambda }}{{\prod }_i{x}_i!},s=\sum _i{x}_i.$$

The factorials are the same under both hypotheses and cancel in the ratio:

$$\Lambda =\frac{L({\lambda }_1)}{L({\lambda }_0)}={\left(\frac{{\lambda }_1}{{\lambda }_0}\right)}^s{e}^{-n({\lambda }_1-{\lambda }_0)},\log \Lambda =s\log \frac{{\lambda }_1}{{\lambda }_0}-n({\lambda }_1-{\lambda }_0).$$

Since ${\lambda }_1>{\lambda }_0$, the coefficient $\log ({\lambda }_1/{\lambda }_0)$ is positive, so $\log \Lambda$ — and hence $\Lambda$ — is a strictly increasing function of $s$ alone. Therefore $\{\Lambda >k\}=\{S>c\}$ with $c=(\log k+n({\lambda }_1-{\lambda }_0))/\log ({\lambda }_1/{\lambda }_0)$. By the Neyman–Pearson lemma the most powerful test of level $\alpha$ rejects when $S$ exceeds the constant that gives level $\alpha$. $■$

(b) The constant $c$ is fixed by the requirement $P(S>c\mid {\lambda }_0)=\alpha$, and under $H_0$ the statistic $S$ is Poisson$(n{\lambda }_0)$. So $c$ depends on ${\lambda }_0$, $n$ and $\alpha$ only — the alternative ${\lambda }_1$ dropped out, exactly as it did for the normal mean. The same rejection region $\{S>c\}$ is therefore most powerful against every ${\lambda }_1>{\lambda }_0$ at once, which is the definition of uniformly most powerful for $H_1:\lambda >{\lambda }_0$.

For ${\lambda }_1<{\lambda }_0$ the coefficient $\log ({\lambda }_1/{\lambda }_0)$ is negative, so $\Lambda$ is strictly decreasing in $s$ and the most powerful test rejects for small $S$. The two-sided alternative contains values on both sides; the best region against one side is an upper tail and against the other a lower tail, and no single region can be both. So no test is most powerful against every $\lambda \ne {\lambda }_0$ simultaneously. $■$

(c) Under $\lambda =1$, $P(X=k)=e^{-1}/k!$, so

$$P(X\le 2)=e^{-1}\left(1+1+\frac{1}{2}\right)=\frac{5}{2}{e}^{-1},P(X\ge 3)=1-\frac{5}{2}{e}^{-1}\approx 1-0.9197=0.080,$$

$$P(X\le 3)=e^{-1}\left(\frac{5}{2}+\frac{1}{6}\right)=\frac{8}{3}{e}^{-1},P(X\ge 4)=1-\frac{8}{3}{e}^{-1}\approx 1-0.9810=\mathrm{0.019.}$$

The level of “reject when $X\ge c$” is $P(X\ge c\mid \lambda =1)$, which is decreasing in $c$ and takes only the discrete values $1,1-e^{-1},1-2e^{-1},0.080,0.019,\dots$. The value $0.05$ is not among them: $c=3$ overshoots and $c=4$ undershoots. The rule with the largest level not exceeding $0.05$ is $c=4$, with level about $0.019$.

Its power against $\lambda =2$: $P(X=k)=2^k{e}^{-2}/k!$, so

$$P(X\le 3\mid \lambda =2)=e^{-2}\left(1+2+2+\frac{4}{3}\right)=\frac{19}{3}{e}^{-2},\text{power}=1-\frac{19}{3}{e}^{-2}\approx 1-0.857=\mathrm{0.143.}$$

One observation carries little information about a rate, and the discreteness has forced the test to run at level $0.019$ rather than the nominal $0.05$, which costs power on top of that.

A remark on the lemma's wording. “$k$ chosen so that the level is exactly $\alpha$” is not achievable with a non-randomised rule when the data are discrete. The exact-level Neyman–Pearson test here rejects when $X\ge 4$, and when $X=3$ rejects with a probability $\gamma$ chosen so that $0.019+\gamma P(X=3\mid \lambda =1)=0.05$; since $P(X=3\mid \lambda =1)=e^{-1}/6\approx 0.0613$, this gives $\gamma \approx 0.51$. In practice one reports the attainable level instead.

(a) Let $G(t)=P(|T|\le t\mid H_0)$ for $t\ge 0$. By symmetry,

$$G(t)=P(-t\le T\le t)=F(t)-F(-t)=2F(t)-1,$$

which is continuous and strictly increasing on the support of $|T|$, from $0$ to $1$. The two-sided $p$-value is $p=P(|T|\ge |t_{\text{obs}}|)=1-G(|t_{\text{obs}}|)=2(1-F(|t_{\text{obs}}|))$, matching the stated formula; as a random variable, $p=1-G(|T|)$.

Now fix $u\in (0,1)$. Because $G$ is continuous and strictly increasing it has an inverse on $(0,1)$, and

$$P(p\le u)=P(G(|T|)\ge 1-u)=P(|T|\ge G^{-1}(1-u))=1-G(G^{-1}(1-u))=u,$$

where the third equality uses continuity, so that $P(|T|=G^{-1}(1-u))=0$ and the inequality may be taken strict or not. A random variable with $P(p\le u)=u$ on $(0,1)$ is uniform on $(0,1)$. $■$

The argument is the one-sided proof applied to the statistic $|T|$: symmetry was used only to write $G$ in terms of $F$. In particular $P(p\le \alpha \mid H_0)=\alpha$, so “reject when $p\le \alpha$” is a level-$\alpha$ test, and it coincides with rejecting when $|T|>G^{-1}(1-\alpha )$, the two-sided critical value.

(b) By definition $z_{\alpha }={\Phi }^{-1}(1-\alpha )$, which is a strictly decreasing function of $\alpha$ because ${\Phi }^{-1}$ is increasing. Then, for $\alpha \in (0,1)$,

$$z_{\text{obs}}>z_{\alpha }⟺\Phi (z_{\text{obs}})>1-\alpha ⟺\alpha >1-\Phi (z_{\text{obs}})=p,$$

applying the increasing function $\Phi$ in the first step and rearranging in the second. So the set of levels at which the data are rejected is exactly the interval $(p,1)$, whose infimum is $p$. $■$

At $\alpha =p$ itself the strict rule $Z>z_{\alpha }$ does not reject — $z_{\text{obs}}=z_p$ exactly — which is why the statement is an infimum rather than a minimum; the distinction has probability zero and is invisible in practice. This is the operational meaning of a $p$-value: it is the level at which the verdict flips, so reporting $p$ tells every reader the verdict at their own $\alpha$.

(c) Let $p$ be the $p$-value of a submitted test. Its null is true, so by the theorem that the $p$-value is uniform under the null (or by (a)), $p\sim \text{Uniform}(0,1)$. Publication is the event $\{p\le 0.05\}$, of probability $0.05$. For $0<u\le 0.05$,

$$P(p\le u\mid p\le 0.05)=\frac{P(p\le u\text{and}p\le 0.05)}{P(p\le 0.05)}=\frac{P(p\le u)}{0.05}=\frac{u}{0.05},$$

and for $u>0.05$ the conditional probability is $1$. This is the distribution function of Uniform$(0,0.05)$. $■$

Hence $P(p<0.01\mid \text{published})=0.01/0.05=\frac{1}{5}$, and likewise $P(0.04<p\le 0.05\mid \text{published})=0.01/0.05=\frac{1}{5}$: each fifth of the interval receives $20\%$ of the published values.

Interpretation. If the published nulls were all true and the filter were the only selection, the published $p$-values would be flat across $(0,0.05)$. If instead some effects were real, their $p$-values would concentrate near $0$ — under a real effect the distribution of $p$ piles up at the low end — and the published histogram would be heavier below $0.01$ than just below $0.05$. Neither honest scenario produces an excess just under the threshold. A pile-up in $(0.04,0.05]$ relative to $(0,0.01)$ is therefore the signature of results that were nudged across the line: choosing among analyses, subgroups, or stopping times until $p$ dipped below $0.05$, which the printed $p$-value does not record.

(a) Let $A_i$ be the event that test $i$ falsely rejects, and $I_i$ its indicator. Since every null is true, $P(A_i)=\alpha$, so each $I_i$ is Bernoulli$(\alpha )$; the tests are independent, so the $I_i$ are independent; and $K={\sum }_{i=1}^m{I}_i$ is a sum of $m$ independent Bernoulli$(\alpha )$ variables, which is Binomial$(m,\alpha )$.

For $m=20$, $\alpha =0.05$: $E[K]=m\alpha =1$ — on average one false finding per study of pure noise — and

$$P(K\ge 1)=1-P(K=0)=1-(1-\alpha {)}^m=1-(0.95{)}^{20}\approx 1-0.3585=0.6415,$$

which is the family-wise error rate under multiplicity.

Smallest $m$ with $P(K\ge 1)\ge \frac{1}{2}$: we need $(0.95{)}^m\le \frac{1}{2}$. Taking logarithms (which reverses the inequality, since $\ln 0.95<0$),

$$m\ln 0.95\le -\ln 2⟺m\ge \frac{\ln 2}{-\ln 0.95}\approx \frac{0.6931}{0.0513}\approx 13.5,$$

so $m=14$. Check: $(0.95{)}^{10}\approx 0.5987$ and $(0.95{)}^3\approx 0.8574$, so $(0.95{)}^{13}\approx 0.513>\frac{1}{2}$ while $(0.95{)}^{14}\approx 0.488<\frac{1}{2}$ ✓. Fourteen honest tests on noise are more likely than not to yield a “finding”.

(b) Bonferroni under arbitrary dependence. With each test at level $\alpha /m$, $P(A_i)\le \alpha /m$ for every true null. By the union bound — subadditivity of probability, which needs no independence —

$$P\left(\bigcup _{i=1}^m{A}_i\right)\le \sum _{i=1}^{m}P(A_i)\le m\cdot \frac{\alpha }{m}=\alpha .■$$

The independent case. By (a) with $\alpha /m$ in place of $\alpha$, the number of false rejections is Binomial$(m,\alpha /m)$ and the exact family-wise error rate is

$$1-{\left(1-\frac{\alpha }{m}\right)}^m.$$

Bernoulli's inequality with $x=\alpha /m\in [0,1]$ gives $(1-\alpha /m{)}^m\ge 1-m\cdot \alpha /m=1-\alpha$, hence

$$1-{\left(1-\frac{\alpha }{m}\right)}^m\le \alpha .■$$

(For completeness, Bernoulli's inequality by induction: it is an equality at $m=1$, and if $(1-x{)}^m\ge 1-mx$ then, multiplying by $1-x\ge 0$, $(1-x{)}^{m+1}\ge (1-mx)(1-x)=1-(m+1)x+m{x}^2\ge 1-(m+1)x$.)

The inequality is strict for $m\ge 2$ and $x>0$, so even independent tests do not use the whole budget: at $m=20$, $\alpha =0.05$ the true rate is $1-(0.9975{)}^{20}\approx 0.049$. The shortfall is small here — of order ${\alpha }^2$ — but it grows when the tests are positively correlated, where the union bound is far from tight and the correction costs power for a guarantee that was already met.

(c) By the post-test probability theorem,

$$P(H_1\mid \text{reject})=\frac{(1-\beta )\pi }{(1-\beta )\pi +\alpha (1-\pi )}.$$

This is at least $\frac{1}{2}$ exactly when the numerator is at least half the denominator, i.e. $2(1-\beta )\pi \ge (1-\beta )\pi +\alpha (1-\pi )$, i.e.

$$(1-\beta )\pi \ge \alpha (1-\pi ),$$

and every step reverses, since the denominator is positive. In words: the rate of true positives must at least match the rate of false positives.

With $\pi =0.05$: the false-positive rate is $\alpha (1-\pi )=0.95\alpha$.

At $\alpha =0.05$: $0.05(1-\beta )\ge 0.05(0.95)$, so the power must be at least $0.95$.

At $\alpha =0.005$: $0.05(1-\beta )\ge 0.005(0.95)=0.00475$, so the power must be at least $0.095$.

Conclusion. In a field that tests long shots — one hypothesis in twenty being real — a significant result at the conventional $5\%$ level is more likely false than true unless the study has $95\%$ power, which almost no study has; and even at perfect power the post-test probability is only $\frac{0.05}{0.05+0.0475}\approx 0.51$. Power cannot rescue such a literature, because the false positives come from the large pool of true nulls, not from the small pool of real effects. Lowering $\alpha$ tenfold shrinks that pool's contribution tenfold, and then even a modest $10\%$ power makes a rejection more likely right than wrong. The $p$-value's meaning depends on two things it does not contain: how plausible the hypothesis was before the data, and how powerful the test was.