Print this page and work through it on paper. Revealed answers are never printed, so you can check one question and still print a clean sheet.

1. 1.Computation · 6 min

   A sample of four observations is to be tested against a fully specified continuous distribution function $F_0$. Evaluated at the ordered observations $x_{(1)}<x_{(2)}<x_{(3)}<x_{(4)}$, the hypothesised distribution function takes the values

   $$F_0(x_{(1)})=0.10,F_0(x_{(2)})=0.45,F_0(x_{(3)})=0.55,F_0(x_{(4)})=\mathrm{0.70.}$$

   1. (a)Compute the Kolmogorov-Smirnov statistic $D_4$, stating at which observation, and from which side of the jump, the supremum is attained.
   2. (b)Suppose instead that $F_0(x_{(4)})=0.97$, the other three values being unchanged. Recompute $D_4$. Which gap is now the largest, and what does this say about checking only one side of each jump?
   3. (c)Without any further calculation, explain why the null distribution of $D_4$ would be the same whether $F_0$ were a normal, an exponential or a logistic distribution function, and name the transformation that shows it.
   Show the solution to question 1

   (a) Both gaps at every jump. The empirical distribution function $F_4$ is a staircase that equals $(i-1)/4$ just before $x_{(i)}$ and $i/4$ just after it. Since $F_0$ is continuous, the largest vertical distance between the two functions occurs at one of these eight values.

   The “after” gaps $|i/4-F_0(x_{(i)})|$ are

   $$|0.25-0.10|=0.15,|0.50-0.45|=0.05,|0.75-0.55|=0.20,|1.00-0.70|=\mathrm{0.30.}$$

   The “before” gaps $|(i-1)/4-F_0(x_{(i)})|$ are

   $$|0-0.10|=0.10,|0.25-0.45|=0.20,|0.50-0.55|=0.05,|0.75-0.70|=\mathrm{0.05.}$$

   The largest of the eight is $0.30$, so $D_4=0.30$, attained just after the fourth observation: the staircase has reached $1$ while $F_0$ is still at $0.70$.

   (b) Move the last point. Only the two gaps at $x_{(4)}$ change. After the jump, $|1.00-0.97|=0.03$; before it, $|0.75-0.97|=0.22$. The other six gaps are as in (a), the largest of them being $0.20$ (twice). So now

   $$D_4=0.22,$$

   attained just before the fourth observation. In (a) the maximum came from the after-side of the last jump; here it comes from the before-side of the same jump, and a computation that checked only the after-gaps would have reported $0.20$ instead of $0.22$. The supremum of $|F_n-F_0|$ is always at a jump, but it can be on either side, so both must be checked at every observation.

   (c) Distribution-free. By the probability integral transform, if $X$ has continuous distribution function $F_0$ then $U=F_0(X)$ is Uniform$(0,1)$. Since $F_0$ is non-decreasing, the empirical distribution function of the transformed values $u_i=F_0(x_i)$, evaluated at $u=F_0(x)$, equals $F_4(x)$; hence

   $$D_4=\underset{x}{\sup }|F_4(x)-F_0(x)|=\underset{u\in (0,1)}{\sup }|G_4(u)-u|,$$

   where $G_4$ is the empirical distribution function of four uniforms. Nothing about the particular $F_0$ survives the substitution, so the null distribution of $D_4$ is the same for every continuous $F_0$ — which is why a single table of critical values serves every goodness-of-fit problem of this type. Notice that the data in (a) were given to you already in the form $F_0(x_{(i)})$: those four numbers are exactly the transformed sample $u_{(i)}$.

2. 2.Computation · 8 min

   Ten patients were randomised, five to a treatment and five to a control, and each was classified as improved or not:

   $$\begin{array}{lccc}& \text{improved}& \text{not}& \\ \text{treatment}& 5& 0& 5\\ \text{control}& 1& 4& 5\\ & 6& 4& 10\end{array}$$

   Write $a$ for the top-left count, so that the observed table has $a=5$.

   1. (a)Compute the four expected counts under independence and explain why the chi-square test of independence should not be used here.
   2. (b)With both margins fixed, list every value $a$ can take, and compute the hypergeometric probability of each possible table. Check that your probabilities sum to $1$.
   3. (c)Give the one-sided $p$-value of Fisher's exact test against the alternative that treatment improves the outcome, and the two-sided $p$-value obtained by summing the probabilities of all tables at least as unlikely as the observed one.
   4. (d)Compute the chi-square statistic anyway (the $5\%$ critical value of ${\chi }_1^2$ is $3.841$) and compare its verdict with the exact two-sided $p$. Which of the two limitations of the chi-square test does the comparison illustrate?
   Show the solution to question 2

   (a) Expected counts. Under independence $E_{ij}=(\text{row total})(\text{column total})/n$, so

   $$E_{11}=\frac{5\times 6}{10}=3,E_{12}=\frac{5\times 4}{10}=2,E_{21}=3,E_{22}=2.$$

   All four are below $5$. The ${\chi }_1^2$ distribution for $X^2$ is asymptotic and needs expected counts of roughly $5$ or more; with $n=10$ the statistic takes only a handful of distinct values, and the continuous approximation cannot be trusted. Fisher's exact test is the right tool.

   (b) The possible tables. The row totals are $5$ and $5$ and the column totals are $6$ and $4$. Since the improved column has total $6$ and each row has only $5$ patients, $a\le 5$ and $6-a\le 5$, so $a\in \{1,2,3,4,5\}$. By the hypergeometric formula with $a+b=5$, $c+d=5$ and $a+c=6$,

   $$P(a)=\frac{\left(\genfrac{}{}{0ex}{}{5}{a}\right)\left(\genfrac{}{}{0ex}{}{5}{6-a}\right)}{\left(\genfrac{}{}{0ex}{}{10}{6}\right)},\left(\genfrac{}{}{0ex}{}{10}{6}\right)=210.$$

   The numerators are

   $$a=1:\left(\genfrac{}{}{0ex}{}{5}{1}\right)\left(\genfrac{}{}{0ex}{}{5}{5}\right)=5,a=2:\left(\genfrac{}{}{0ex}{}{5}{2}\right)\left(\genfrac{}{}{0ex}{}{5}{4}\right)=50,a=3:\left(\genfrac{}{}{0ex}{}{5}{3}\right)\left(\genfrac{}{}{0ex}{}{5}{3}\right)=100,a=4:\left(\genfrac{}{}{0ex}{}{5}{4}\right)\left(\genfrac{}{}{0ex}{}{5}{2}\right)=50,a=5:\left(\genfrac{}{}{0ex}{}{5}{5}\right)\left(\genfrac{}{}{0ex}{}{5}{1}\right)=5,$$

   and $5+50+100+50+5=210$, so the probabilities $\frac{5}{210},\frac{50}{210},\frac{100}{210},\frac{50}{210},\frac{5}{210}$ sum to $1$. ✓ The observed table has probability $\frac{5}{210}=\frac{1}{42}$.

   (c) The $p$-values. “Treatment improves the outcome” means large $a$. The observed $a=5$ is the largest possible value, so the only table at least as extreme is the observed one:

   $$p_{\text{one-sided}}=\frac{1}{42}\approx \mathrm{0.024.}$$

   For the two-sided test, the tables with probability at most $\frac{5}{210}$ are $a=5$ and $a=1$ (the mirror image, in which the control group does all the improving). So

   $$p_{\text{two-sided}}=\frac{5+5}{210}=\frac{10}{210}=\frac{1}{21}\approx \mathrm{0.048.}$$

   (d) The approximation, for comparison. With the observed counts $5,0,1,4$ and expected counts $3,2,3,2$,

   $$X^2=\frac{(5-3{)}^2}{3}+\frac{(0-2{)}^2}{2}+\frac{(1-3{)}^2}{3}+\frac{(4-2{)}^2}{2}=\frac{4}{3}+2+\frac{4}{3}+2=\frac{20}{3}\approx 6.67,$$

   which is well beyond $3.841$ and corresponds to a $p$-value near $0.01$. The exact two-sided $p$ is $0.048$ — nearly five times larger. This is the small expected counts limitation: with a discrete, skewed statistic the ${\chi }^2$ approximation rejects too often, exactly as the rule of thumb warns. (The other limitation, arbitrariness of binning, does not arise because the data are genuinely categorical.) Note what not to do: there is nothing here to pool, and pooling would change the hypothesis anyway.

3. 3.Computation · 7 min

   Two unrelated studies. Use ${\chi }_{0.05,4}^2=9.488$ and ${\chi }_{0.05,1}^2=3.841$.

   Study I. A $3\times 3$ table on $n=400$ subjects gives a chi-square statistic of $X^2=32$.

   Study II. Two hundred people were asked the same yes/no question before and after a campaign. Seven changed from no to yes, one changed from yes to no, and the remaining $192$ gave the same answer both times.

   1. (a)For Study I, test independence at $\alpha =0.05$ and compute Cramér's $V$. Then find $X^2$ and $V$ for a table with the identical pattern of proportions but $n=100$, and say in one sentence what the comparison shows.
   2. (b)For Study II, carry out McNemar's test at $\alpha =0.05$. Then compute the exact two-sided $p$-value by treating the $8$ changers as a Binomial$(8,\frac{1}{2})$ count, and reconcile the two verdicts.
   3. (c)Explain why the $192$ unchanged subjects in Study II carry no information about whether the campaign shifted opinion, and say what a chi-square test of independence on the $2\times 2$ before-by-after table would be testing instead.
   Show the solution to question 3

   (a) Study I. Degrees of freedom $(3-1)(3-1)=4$, and $32>9.488$, so independence is rejected at the $5\%$ level (the $p$-value is far below $0.001$). For the strength, $\min (r-1,c-1)=2$, so

   $$V=\sqrt{\frac{X^2}{n\min (r-1,c-1)}}=\sqrt{\frac{32}{400\times 2}}=\sqrt{0.04}=\mathrm{0.2.}$$

   For a fixed pattern of association $X^2$ is proportional to $n$, so at $n=100$ the statistic is $X^2=32\times \frac{100}{400}=8$, which is below $9.488$: not significant. But

   $$V=\sqrt{\frac{8}{100\times 2}}=\sqrt{0.04}=0.2$$

   exactly as before. The statistic measures evidence and tracks the sample size; $V$ measures strength and does not — and $V=0.2$ is a weak association whether or not $n$ is large enough to make it significant.

   (b) Study II. With $b=7$ and $c=1$,

   $$X^2=\frac{(b-c{)}^2}{b+c}=\frac{36}{8}=4.5>3.841,$$

   so McNemar's test rejects at the $5\%$ level.

   Exactly: under $H_0$ each changer is equally likely to change in either direction, so conditional on $b+c=8$ changers, $b\sim \text{Binomial}(8,\frac{1}{2})$. The one-sided tail is

   $$P(b\ge 7)=\frac{\left(\genfrac{}{}{0ex}{}{8}{7}\right)+\left(\genfrac{}{}{0ex}{}{8}{8}\right)}{2^8}=\frac{8+1}{256}=\frac{9}{256},$$

   and doubling gives the two-sided $p=\frac{18}{256}=\frac{9}{128}\approx 0.070$ — not significant at $5\%$.

   The verdicts differ because ${\chi }_1^2$ is an asymptotic approximation to a statistic that, with only $8$ discordant pairs, takes very few values; the expected count of each kind of change is only $4$, below the working threshold of $5$. The exact binomial calculation is the one to trust. The chi-square version is the same statistic with a continuous distribution stretched over nine possible outcomes, and it rejects too readily, as approximations to discrete counts do.

   (c) Why the concordant pairs drop out. The question is whether the marginal proportion of yes answers moved. A subject who answered the same way twice contributes one yes (or one no) to both margins and therefore cannot move the difference between them, whichever way the campaign worked. Under the null the only random quantity is the direction of each change, and that is what the test conditions on — the same logic as pairing in the paired $t$-test, where within-pair differences discard between-subject variation.

   A chi-square test of independence on the before-by-after table would ask a different question: whether a person's before answer is associated with their after answer. With $192$ of $200$ unchanged it would reject overwhelmingly, reporting only that people tend to agree with themselves — which says nothing about whether the campaign shifted opinion.

4. 4.Computation · 8 min

   Eight subjects were measured before and after a treatment. The after-minus-before differences are

   $$+2.4,+0.7,-0.3,+5.1,+1.9,+3.3,+0.8,+\mathrm{4.0.}$$

   Use $z_{0.025}=1.96$ where a normal approximation is asked for.

   1. (a)Carry out the sign test of $H_0$: median difference is zero, giving the exact one-sided and two-sided $p$-values. State the only assumption the test needs.
   2. (b)Compute the Wilcoxon signed-rank statistic $W^{+}$, its null mean and variance, and the standardised value; decide at the $5\%$ two-sided level. Show without a calculator that your decision does not depend on the rounding of the square root.
   3. (c)Find the exact two-sided $p$-value of the signed-rank test by counting sign patterns, and explain why the two tests disagree on these data.
   Show the solution to question 4

   (a) Sign test. Seven differences are positive and one is negative, no ties. Under $H_0$ the number of positives is $S\sim \text{Binomial}(8,\frac{1}{2})$, which needs only that each difference is positive with probability $\frac{1}{2}$ — no symmetry, no shape. Then

   $$P(S\ge 7)=\frac{\left(\genfrac{}{}{0ex}{}{8}{7}\right)+\left(\genfrac{}{}{0ex}{}{8}{8}\right)}{2^8}=\frac{9}{256}\approx 0.035,$$

   so the two-sided $p$-value is $\frac{18}{256}=\frac{9}{128}\approx 0.070$. Not significant at the $5\%$ level.

   (b) Signed-rank test. Order the absolute differences and rank them:

   $$0.3(1),0.7(2),0.8(3),1.9(4),2.4(5),3.3(6),4.0(7),5.1(8).$$

   The only negative difference is $-0.3$, with rank $1$, so $W^{-}=1$ and, since the ranks sum to $\frac{8\cdot 9}{2}=36$,

   $$W^{+}=36-1=35.$$

   By the null moments of the signed-rank statistic with $n=8$,

   $$E(W^{+})=\frac{8\cdot 9}{4}=18,\mathrm{Var}(W^{+})=\frac{8\cdot 9\cdot 17}{24}=\frac{1224}{24}=51.$$

   So $z=\frac{35-18}{\sqrt{51}}=\frac{17}{\sqrt{51}}$. Since $7^2=49<51<51.84=7.2^2$, the standard deviation lies between $7$ and $7.2$, hence

   $$2.36<\frac{17}{7.2}<z<\frac{17}{7}<2.43,$$

   and every value in that range exceeds $1.96$. Reject $H_0$ at the $5\%$ level; $z\approx 2.38$, two-sided $p\approx 0.017$.

   (c) Exact $p$. Under the null (differences symmetric about $0$) each of the $2^8=256$ sign patterns is equally likely. $W^{+}\ge 35$ means $W^{-}\le 1$, which happens for exactly two patterns: no negatives ($W^{-}=0$) or the rank-$1$ difference alone negative ($W^{-}=1$). So

   $$P(W^{+}\ge 35)=\frac{2}{256}=\frac{1}{128},p_{\text{two-sided}}=\frac{2}{128}=\frac{1}{64}\approx 0.016,$$

   in close agreement with the normal approximation.

   The two tests disagree because the sign test sees only that $7$ of $8$ differences are positive, and with $n=8$ that pattern is simply not rare enough ($p=0.070$). The signed-rank test also sees that the one negative difference is the smallest in magnitude while the positives include all the large ones. That extra information is what pushes the evidence past the $5\%$ line — and it is bought with the extra assumption of symmetry, which the sign test does not need.

5. 5.Proof · 10 min

   Let $d_1,\dots ,d_n$ be paired differences with no ties and none equal to zero, and let $W^{+}$ be the Wilcoxon signed-rank statistic. You may quote ${\sum }_{i=1}^{n}i=\frac{n(n+1)}{2}$ and ${\sum }_{i=1}^n{i}^2=\frac{n(n+1)(2n+1)}{6}$.

   1. (a)Write $W^{+}={\sum }_{i=1}^{n}i{Z}_i$ for suitable indicator variables $Z_i$ and prove that under the null hypothesis that the differences are symmetric about zero, $E(W^{+})=\frac{n(n+1)}{4}$ and $\mathrm{Var}(W^{+})=\frac{n(n+1)(2n+1)}{24}$. Say precisely where symmetry is used and which step would fail without it.
   2. (b)Prove that under the same null, $W^{-}=\frac{n(n+1)}{2}-W^{+}$ has exactly the same distribution as $W^{+}$, and deduce that $P(W^{+}\ge w)=P\left(W^{+}\le \frac{n(n+1)}{2}-w\right)$ for every $w$.
   3. (c)Using (b), find the smallest attainable one-sided and two-sided $p$-values for a sample of $n=5$ pairs, and hence the smallest $n$ for which a two-sided signed-rank test can ever reject at $\alpha =0.05$.
   Show the solution to question 5

   (a) The moments. Rank the absolute differences $|d_i|$ from $1$ to $n$ and let $Z_i=1$ if the difference holding absolute rank $i$ is positive, $Z_i=0$ otherwise. Then $W^{+}$, the sum of the ranks of the positive differences, is ${\sum }_{i=1}^{n}i{Z}_i$.

   Under symmetry about zero, each difference is equally likely to be positive or negative, and its sign is independent of its magnitude: symmetry says the conditional distribution of the sign given $|d_i|$ is still a fair coin. Hence the signs are independent of the ranking, and the $Z_i$ are independent Bernoulli$(\frac{1}{2})$ variables. This is the step that needs symmetry. Under a skewed null with median zero the signs are still fair coins, but they are no longer independent of the magnitudes — large $|d|$ would be more often positive, say — so the $Z_i$ attached to large ranks would not be Bernoulli$(\frac{1}{2})$, and neither formula would hold.

   By linearity,

   $$E(W^{+})=\sum _{i=1}^{n}iE(Z_i)=\frac{1}{2}\sum _{i=1}^{n}i=\frac{1}{2}\cdot \frac{n(n+1)}{2}=\frac{n(n+1)}{4}.$$

   By independence the variance of the sum is the sum of the variances, and $\mathrm{Var}(Z_i)=\frac{1}{2}\cdot \frac{1}{2}=\frac{1}{4}$, so

   $$\mathrm{Var}(W^{+})=\sum _{i=1}^n{i}^2\mathrm{Var}(Z_i)=\frac{1}{4}\cdot \frac{n(n+1)(2n+1)}{6}=\frac{n(n+1)(2n+1)}{24}.■$$

   Linearity did not need independence; the variance step did.

   (b) Symmetry of the null distribution. The ranks of the negative differences are those $i$ with $Z_i=0$, so

   $$W^{-}=\sum _{i=1}^{n}i(1-Z_i)=\frac{n(n+1)}{2}-W^{+}.$$

   Under the null the vector $(Z_1,\dots ,Z_n)$ consists of independent fair coins, and so does $(1-Z_1,\dots ,1-Z_n)$: the two vectors have the same joint distribution, namely uniform over the $2^n$ sign patterns. Since $W^{+}$ is the function $\sum i{Z}_i$ of the first vector and $W^{-}$ is the same function of the second, $W^{-}$ and $W^{+}$ have the same distribution. Therefore, for any $w$,

   $$P(W^{+}\ge w)=P(W^{-}\ge w)=P\left(\frac{n(n+1)}{2}-W^{+}\ge w\right)=P\left(W^{+}\le \frac{n(n+1)}{2}-w\right).■$$

   The null distribution is symmetric about $\frac{n(n+1)}{4}$, which is why one tail of a table suffices.

   (c) The floor. With $n=5$ the ranks sum to $15$ and there are $2^5=32$ equally likely sign patterns. The most extreme value $W^{+}=15$ arises from exactly one pattern (all positive), so the smallest one-sided $p$-value is $\frac{1}{32}=0.03125$. By (b) the lower tail $W^{+}\le 0$ has the same probability, so the smallest two-sided $p$-value is $\frac{2}{32}=\frac{1}{16}=0.0625$, which exceeds $0.05$: no two-sided signed-rank test on five pairs can reject at the $5\%$ level, whatever the data.

   In general the smallest two-sided $p$ is $\frac{2}{2^n}$, and $\frac{2}{2^n}\le 0.05$ requires $2^n\ge 40$, that is $n\ge 6$: with six pairs the extreme pattern gives $p=\frac{2}{64}=0.03125$. (One-sided, $2^n\ge 20$ suffices, so $n=5$ is enough.) A rank test cannot be more decisive than the number of sign patterns allows, and with tiny samples that ceiling binds before any question about power arises.

6. 6.Critique · 9 min

   Two independent samples of sizes $n_1=5$ and $n_2=7$ are ranked together, and the rank sum of the first sample is $W=17$. A student writes the following. Use $z_{0.025}=1.96$.

   Step 1. The Mann-Whitney statistic is the rank sum, so $U=17$.

   Step 2. $E(U)=\frac{5\times 7}{2}=17.5$ and $\mathrm{Var}(U)=\frac{5\times 7\times 13}{12}=\frac{455}{12}$, so $z=\frac{17-17.5}{\sqrt{455/12}}\approx -0.08$.

   Step 3. Since $|z|<1.96$ we do not reject, and therefore the two populations have the same median.

   Take $U=W-\frac{n_1(n_1+1)}{2}$; this is the number of pairs $(i,j)$ in which the first-sample observation $x_i$ exceeds the second-sample observation $y_j$.

   1. (a)Show that $W-\frac{n_1(n_1+1)}{2}$ does count the pairs described, by considering the $k$-th smallest first-sample observation and its rank in the combined ordering.
   2. (b)Identify every error in the student's argument — there are three, and they are of different kinds — and say which result or hypothesis each one abuses.
   3. (c)Redo the test correctly. Show without a calculator whether $|z|$ exceeds $1.96$, and state the conclusion in a form that is actually justified by the test.
   Show the solution to question 6

   (a) The identity. Let the first-sample observations in increasing order have combined ranks $r_{(1)}<r_{(2)}<\cdots <r_{(n_1)}$. The $k$-th smallest first-sample value has $k-1$ first-sample values below it, so the remaining $r_{(k)}-k$ observations below it all belong to the second sample: $r_{(k)}-k$ is the number of $y_j$ it exceeds. Summing over $k$,

   $$\#\{(i,j):x_i>y_j\}=\sum _{k=1}^{n_1}\left(r_{(k)}-k\right)=W-\frac{n_1(n_1+1)}{2}.■$$

   Its range is $0$ to $n_1{n}_2$, and $W$ alone ranges from $\frac{n_1(n_1+1)}{2}$ upward, which is exactly why the offset matters.

   (b) Three errors.

   Step 1 confuses $W$ with $U$. By the definition of the Mann-Whitney statistic, $U=W-\frac{n_1(n_1+1)}{2}=17-15=2$, not $17$. The null moments $E(U)=\frac{n_1{n}_2}{2}$ and $\mathrm{Var}(U)=\frac{n_1{n}_2(n_1+n_2+1)}{12}$ are moments of $U$; feeding them $W$ compares a statistic with the wrong centre. (Note that $W=17$ is only $2$ above its minimum of $15$: the first sample holds nearly all the lowest ranks, which is the opposite of “no evidence”.)

   Step 3 turns non-rejection into acceptance. Failing to reject $H_0$ means the data are compatible with $H_0$; it never establishes it. With $n_1=5$ and $n_2=7$ the test has modest power, so a non-significant result would be weak evidence of anything.

   Step 3 also misstates the hypothesis. The Mann-Whitney null is that the two distributions are identical, and the alternative is a stochastic ordering, $P(X<Y)\ne \frac{1}{2}$. A conclusion about medians is valid only under the extra assumption that the two distributions differ by a pure shift and are otherwise the same shape — an assumption the student never stated.

   (c) The correct test. With $U=2$,

   $$z=\frac{2-17.5}{\sqrt{455/12}}=\frac{-15.5}{\sqrt{455/12}}.$$

   Compare squares: $|z|>1.96$ if and only if $15.5^2>1.96^2\times \frac{455}{12}$. The left side is $240.25$. On the right, $1.96^2=3.8416$ and $\frac{455}{12}<38$, so the right side is below $3.8416\times 38=145.98$. Since $240.25>146$, $|z|>1.96$ comfortably; in fact $6.1^2=37.21<\frac{455}{12}<38.44=6.2^2$ gives $2.50<|z|<2.54$, so $z\approx -2.52$ and the two-sided $p$ is about $0.012$.

   Reject $H_0$ at the $5\%$ level. The justified conclusion: the two samples are unlikely to come from the same distribution, and the direction is that first-sample values tend to be smaller — only $2$ of the $35$ pairs have the first-sample value on top. Any statement about medians requires adding, and stating, the shift assumption.

7. 7.Computation · 8 min

   Five students sit two examinations. Ranked by the first examination they are students $1,2,3,4,5$; their ranks on the second examination, listed in the same order, are $3,1,2,5,4$. The raw second-examination scores were $y=6,1,3,1000,40$.

   1. (a)Compute Spearman's ${\rho }_s$ from the computing formula, giving the answer as an exact fraction.
   2. (b)List every pair of students as concordant or discordant and compute Kendall's $\tau$. Compare it with ${\rho }_s$.
   3. (c)The score $1000$ is a data-entry error; the true value was $100$. Determine, with reasons, what happens to ${\rho }_s$ and $\tau$ when it is corrected, and explain what happens to Pearson's $r$ computed on the raw scores. Which single feature of the rank definitions is responsible for the difference?
   Show the solution to question 7

   (a) Spearman. With $r_i=1,2,3,4,5$ and $s_i=3,1,2,5,4$, the rank differences $d_i=r_i-s_i$ are

   $$-2,+1,+1,-1,+1,\text{so}\sum d_i^2=4+1+1+1+1=8.$$

   There are no ties, so the computing formula applies with $n(n^2-1)=5\times 24=120$:

   $${\rho }_s=1-\frac{6\times 8}{120}=1-\frac{48}{120}=1-\frac{2}{5}=\frac{3}{5}.$$

   (b) Kendall. Listing the second-exam ranks in first-exam order, $3,1,2,5,4$, a pair $(i,j)$ with $i<j$ is concordant when the second rank increases too. The $\left(\genfrac{}{}{0ex}{}{5}{2}\right)=10$ pairs:

   $$(1,2)3>1D,(1,3)3>2D,(1,4)3<5C,(1,5)3<4C,(2,3)1<2C,$$

   $$(2,4)1<5C,(2,5)1<4C,(3,4)2<5C,(3,5)2<4C,(4,5)5>4D.$$

   So $C=7$, $D=3$ and

   $$\tau =\frac{C-D}{\left(\genfrac{}{}{0ex}{}{5}{2}\right)}=\frac{4}{10}=\frac{2}{5}.$$

   As usual $|\tau |<|{\rho }_s|$ for the same data; here $\tau =\frac{2}{3}{\rho }_s$ exactly, the rough rule of thumb landing on the nose. The interpretation of $\tau$ is direct: a randomly chosen pair of students is ordered the same way by both exams with probability $0.7$ and the opposite way with probability $0.3$, and $\tau$ is the difference.

   (c) Correcting the error. With the true score the second-exam values are $6,1,3,100,40$. Since $100$ is still the largest, student $4$ still has rank $5$ and every other rank is unchanged: the rank vector is $3,1,2,5,4$ exactly as before. Every $d_i$ is unchanged, so ${\rho }_s=\frac{3}{5}$ still; every pair is concordant or discordant exactly as before, so $\tau =\frac{2}{5}$ still. A rank depends only on order, and correcting $1000$ to $100$ preserves the order of the five scores. The same is true of any strictly increasing transformation of the scores — squaring them, taking logarithms — which is why rank correlation detects any monotone relationship and not merely a linear one.

   Pearson's $r$ is built from products of deviations, $S_{xy}=\sum (x_i-\bar{x})(y_i-\bar{y})$, and with $y_4=1000$ that single term dominates $S_{xy}$ while $(y_4-\bar{y}{)}^2$ dominates $S_{yy}$. In the limit of an ever larger $y_4$, $r$ tends to the correlation between $x$ and the indicator of student $4$: the $x$-deviations are $-2,-1,0,1,2$, the indicator's deviations from its mean $0.2$ are $-0.2,-0.2,-0.2,0.8,-0.2$, the cross-products sum to $0.4+0.2+0+0.8-0.4=1$, and with $S_{xx}=10$ and $S_{II}=4(0.04)+0.64=0.8$ the limit is $r=\frac{1}{\sqrt{8}}=\frac{1}{2\sqrt{2}}\approx 0.35$. So $r$ is being reported by one observation, and correcting that observation to $100$ moves $r$ appreciably, while a second error putting it at $10^6$ would move it again. The rank statistics bound the influence of any one observation — the erroneous score can never contribute more than a rank difference of $4$ — and that robustness is a different property from being assumption-free.

8. 8.Proof · 10 min

   Let $r_1,\dots ,r_n$ and $s_1,\dots ,s_n$ be the ranks of two variables on the same $n$ observations, with no ties, and let $d_i=r_i-s_i$. You may quote ${\sum }_{i=1}^{n}i=\frac{n(n+1)}{2}$ and ${\sum }_{i=1}^n{i}^2=\frac{n(n+1)(2n+1)}{6}$, but not the identity in (a).

   1. (a)Prove that $\sum _{i=1}^n{\left(i-\frac{n+1}{2}\right)}^2=\frac{n(n^2-1)}{12}$.
   2. (b)Starting from the definition of ${\rho }_s$ as Pearson's correlation of the two rank vectors, derive the computing formula ${\rho }_s=1-\frac{6\sum d_i^2}{n(n^2-1)}$. Say where the absence of ties is used.
   3. (c)Prove that if the second ranking is the exact reverse of the first, $s_i=n+1-r_i$ for every $i$, then ${\rho }_s=-1$. Deduce that $\sum d_i^2\le \frac{n(n^2-1)}{3}$ for any two rankings, and check both statements by hand for $n=3$.
   Show the solution to question 8

   (a) The sum of squared deviations of $1,\dots ,n$. Expand the square and use the two quoted sums:

   $$\sum _{i=1}^n{\left(i-\frac{n+1}{2}\right)}^2=\sum i^2-(n+1)\sum i+n\cdot \frac{(n+1{)}^2}{4}=\frac{n(n+1)(2n+1)}{6}-\frac{n(n+1{)}^2}{2}+\frac{n(n+1{)}^2}{4}.$$

   The last two terms combine to $-\frac{n(n+1{)}^2}{4}$. Factor out $n(n+1)$ and put everything over $12$:

   $$n(n+1)\left[\frac{2(2n+1)-3(n+1)}{12}\right]=n(n+1)\cdot \frac{4n+2-3n-3}{12}=\frac{n(n+1)(n-1)}{12}=\frac{n(n^2-1)}{12}.■$$

   (b) The computing formula. Because there are no ties, each rank vector is a permutation of $1,\dots ,n$. So both have the same mean $\bar{r}=\bar{s}=\frac{n+1}{2}$ and, by (a), the same sum of squared deviations,

   $$S_{rr}=S_{ss}=\frac{n(n^2-1)}{12}.$$

   (With ties, averaged ranks would spoil both of these — the means would still agree but the sums of squares would be smaller, and the formula would need a correction.)

   Now write $d_i=(r_i-\bar{r})-(s_i-\bar{s})$, which is legitimate because the two means are equal, and expand:

   $$\sum d_i^2=\sum (r_i-\bar{r}{)}^2-2\sum (r_i-\bar{r})(s_i-\bar{s})+\sum (s_i-\bar{s}{)}^2=S_{rr}-2S_{rs}+S_{ss}=2S_{rr}-2S_{rs}.$$

   Hence $S_{rs}=S_{rr}-\frac{1}{2}\sum d_i^2$, and Pearson's correlation of the ranks is

   $${\rho }_s=\frac{S_{rs}}{\sqrt{S_{rr}{S}_{ss}}}=\frac{S_{rr}-\frac{1}{2}\sum d_i^2}{S_{rr}}=1-\frac{\sum d_i^2}{2S_{rr}}=1-\frac{\sum d_i^2}{2\cdot n(n^2-1)/12}=1-\frac{6\sum d_i^2}{n(n^2-1)}.■$$

   (c) Reversed rankings. If $s_i=n+1-r_i$ then

   $$d_i=r_i-(n+1-r_i)=2r_i-(n+1)=2\left(r_i-\frac{n+1}{2}\right),$$

   so, since the $r_i$ are a permutation of $1,\dots ,n$, part (a) gives

   $$\sum d_i^2=4\sum _{i=1}^n{\left(i-\frac{n+1}{2}\right)}^2=4\cdot \frac{n(n^2-1)}{12}=\frac{n(n^2-1)}{3},$$

   and the computing formula returns ${\rho }_s=1-\frac{6}{n(n^2-1)}\cdot \frac{n(n^2-1)}{3}=1-2=-1$. $■$

   For the bound: ${\rho }_s$ is a Pearson correlation, so by the Cauchy–Schwarz inequality ${\rho }_s\ge -1$ for any two rankings. Substituting the computing formula, $1-\frac{6\sum d_i^2}{n(n^2-1)}\ge -1$ rearranges to $\sum d_i^2\le \frac{n(n^2-1)}{3}$, with equality exactly when ${\rho }_s=-1$ — which the reversed ranking attains. (Equivalently: reversing the order minimises $\sum r_i{s}_i$, the rearrangement inequality in rank form.)

   Check, $n=3$. The bound is $\frac{3\cdot 8}{3}=8$. Reversing $1,2,3$ to $3,2,1$ gives $d=-2,0,2$ and $\sum d_i^2=4+0+4=8$, so ${\rho }_s=1-\frac{6\cdot 8}{24}=-1$. ✓ The other five permutations give $\sum d_i^2\in \{0,2,2,6,6\}$, all below $8$, so the bound is attained only by the reversal. ✓

9. 9.Sketch · 9 min

   Two groups give $x=1,2$ and $y=4,6,9$. A permutation test is to be run on $T=\bar{y}-\bar{x}$, one-sided against $\bar{y}>\bar{x}$. Write $s$ for the sum of the two values assigned to the first group.

   1. (a)Show that $T=\frac{44-5s}{6}$, and deduce that rejecting for large $T$ is the same as rejecting for small $s$. Why is $s$ the better statistic to tabulate?
   2. (b)List all $\left(\genfrac{}{}{0ex}{}{5}{2}\right)=10$ relabellings with their values of $s$, verify that the average of the ten values equals $n_1$ times the pooled mean, and draw the permutation null distribution of $s$ as a bar chart, marking the observed value and shading the rejection region that the observed data define.
   3. (c)Read the one-sided $p$-value off your chart. Then say what is the two-sided $p$-value under the rule “at least as far from $E(s)$ as the observed value”, and explain from the chart why this distribution, unlike the equal-sizes case, is not symmetric.
   Show the solution to question 9

   (a) Reduce to the group sum. The pooled total is $1+2+4+6+9=22$. If the first group's two values sum to $s$, the second group's three values sum to $22-s$, so

   $$T=\bar{y}-\bar{x}=\frac{22-s}{3}-\frac{s}{2}=\frac{2(22-s)-3s}{6}=\frac{44-5s}{6}.$$

   This is a strictly decreasing function of $s$, so “$T$ at least as large as observed” is exactly “$s$ at least as small as observed”, and the two tests have identical $p$-values. Tabulating $s$ is better because it takes integer values, and because the argument shows that any statistic that is a monotone function of the first group's sum — a difference of totals, of means, a $t$-statistic with pooled variance — yields the same test.

   (b) The ten relabellings. Choosing which two of $\{1,2,4,6,9\}$ go to the first group:

   $$\{1,2\}\to 3,\{1,4\}\to 5,\{2,4\}\to 6,\{1,6\}\to 7,\{2,6\}\to 8,\{1,9\}\to 10,\{4,6\}\to 10,\{2,9\}\to 11,\{4,9\}\to 13,\{6,9\}\to 15.$$

   The observed labelling is $\{1,2\}$, so $s_{\text{obs}}=3$, the smallest value in the list. The ten values sum to $3+5+6+7+8+10+10+11+13+15=88$, so their average is $8.8$; the pooled mean is $22/5=4.4$ and $n_1\times 4.4=8.8$. ✓ (Under exchangeability each of the two first-group slots holds a randomly chosen pooled value, so $E(s)=2\times$ pooled mean.)

   The chart. Horizontal axis $s$ from $3$ to $15$ with integer ticks; vertical axis “number of relabellings” from $0$ to $2$. Draw a bar of height $1$ at each of $s=3,5,6,7,8,11,13,15$ and a bar of height $2$ at $s=10$: nine bars, total height $10$. Mark the bar at $s=3$ as the observed value — it stands alone at the far left. The rejection region defined by the data is $\{s\le 3\}$, which is just that one bar, so shade it. Draw a vertical dashed line at $E(s)=8.8$; it falls between the bars at $8$ and $10$.

   (c) Reading off. Exactly one of the ten equally likely relabellings has $s\le 3$, so

   $$p_{\text{one-sided}}=\frac{1}{10}.$$

   This is also the smallest $p$-value the design can produce: with $n_1=2$ and $n_2=3$ nothing can reach $\alpha =0.05$, however cleanly the groups separate.

   For the two-sided rule, the observed distance is $|3-8.8|=5.8$, so a relabelling counts if $s\le 3$ or $s\ge 14.6$. From the chart that is $s=3$ and $s=15$, two of ten: $p_{\text{two-sided}}=\frac{2}{10}$.

   Why no symmetry. In the chapter's equal-sizes example every relabelling has a mirror image obtained by swapping the two groups, which negates $T$, so the null distribution is symmetric about $0$. Here the groups have sizes $2$ and $3$, and swapping them is not a relabelling of the same design, so nothing forces symmetry. The chart shows it: the bars to the left of $8.8$ sit at distances $5.8,3.8,2.8,1.8,0.8$ while those to the right sit at $1.2,1.2,2.2,4.2,6.2$, and the value $9$ stretches the right tail. Exactness does not need symmetry — the proof that the permutation test has the stated level uses only that the ten relabellings are equally likely — but reading a two-sided $p$-value from an asymmetric distribution does require saying which rule is being used.

10. 10.Critique · 9 min

    Two arguments follow. Each contains exactly one fatal step. For each argument, name the step, explain why it fails, and give the corrected procedure.

    Argument A. “Five observations $0.9,2.3,3.1,4.4,4.8$ come from a Uniform$(0,\theta )$ population. Step 1. The maximum likelihood estimate of $\theta$ is the sample maximum, $4.8$. Step 2. The sampling distribution of the maximum is awkward, so bootstrap it: draw resamples of size $5$ with replacement, compute the maximum of each, and take the $2.5\%$ and $97.5\%$ quantiles of the resample maxima as a $95\%$ confidence interval for $\theta$. Step 3. Since the interval comes from resampling, it makes no assumption about the population.”

    Argument B. “Ten paired differences are $-0.3,-0.1,0.2,0.4,0.5,0.7,0.8,9,12,15$. Step 1. The differences are strongly right-skewed, so the paired $t$-test is unreliable. Step 2. Rank tests are distribution-free, so the Wilcoxon signed-rank test is valid whatever the shape of the differences. Step 3. Apply it to test whether the median difference is zero.”

    1. (a)For Argument A, compute exactly the probability that a resample's maximum equals the observed maximum $4.8$, and use it to describe the shape of the bootstrap distribution of the maximum. What does that shape do to the upper end of the percentile interval, and why is that fatal for a parameter that is certainly at least $4.8$?
    2. (b)For Argument B, say what the Wilcoxon null actually assumes beyond a zero median, and show concretely on these ten differences why the test is not what its user thinks it is. What should be used instead, and what does “distribution-free” actually promise?
    Show the solution to question 10

    (a) Argument A: Step 2 is the fatal one. Step 1 is correct, and Step 3 is only wrong as a consequence of Step 2.

    The resample maximum is at most the observed maximum, because a resample contains only observed values. It equals $4.8$ unless all five draws miss the largest observation, and each draw misses it with probability $\frac{4}{5}$ independently, so

    $$P(\text{resample max}=4.8)=1-{\left(\frac{4}{5}\right)}^5=1-\frac{1024}{3125}=\frac{2101}{3125}\approx \mathrm{0.672.}$$

    So the bootstrap distribution of the maximum puts about two-thirds of its mass on the single point $4.8$ and the rest at $4.4$, $3.1$, $2.3$ or $0.9$ — a distribution that lives entirely at or below the estimate. Its $97.5\%$ quantile is therefore $4.8$ itself (indeed every quantile above the $33\%$ point is $4.8$), so the percentile interval has the form $(\text{something},4.8]$.

    That is fatal because $\theta \ge 4.8$ with certainty — the population cannot produce a value larger than $\theta$ — and $\theta >4.8$ with probability one. An interval whose upper endpoint is $4.8$ contains $\theta$ with probability zero, not $0.95$. The bootstrap fails here because the statistic depends on an extreme order statistic, and the empirical distribution is a poor stand-in for the population exactly at its boundary. (As $n$ grows the defect does not go away: $1-(1-\frac{1}{n}{)}^n\to 1-e^{-1}\approx 0.63$.)

    Correction. The sampling distribution of the maximum is not awkward at all: for a Uniform$(0,\theta )$ sample of size $n$, $P(X_{(n)}\le t)=(t/\theta {)}^n$ for $0\le t\le \theta$. Use that exact distribution to build the interval, as the estimation of $\theta$ for the uniform is done in closed form; the bootstrap is the wrong tool for a boundary parameter. Step 3's claim of “no assumption” is also empty: the argument already assumed a uniform population to justify the maximum in Step 1.

    (b) Argument B: Step 2 is the fatal one. Step 1 is a fair observation, and Step 3 merely acts on Step 2.

    The null hypothesis of the Wilcoxon signed-rank test is that the differences are symmetric about zero, not merely that their median is zero. The null distribution of $W^{+}$ — $E(W^{+})=\frac{n(n+1)}{4}$, $\mathrm{Var}(W^{+})=\frac{n(n+1)(2n+1)}{24}$, and the uniform distribution over the $2^n$ sign patterns — is derived by writing $W^{+}=\sum i{Z}_i$ with the signs $Z_i$ independent fair coins independent of the magnitudes. Symmetry is what delivers that independence.

    On these data the assumption is visibly false. Ranking the absolute values gives $0.1(1),0.2(2),0.3(3),0.4(4),0.5(5),0.7(6),0.8(7),9(8),12(9),15(10)$; the two negatives hold ranks $1$ and $3$, so $W^{-}=4$ and $W^{+}=55-4=51$, against a null mean of $27.5$ and null standard deviation $\sqrt{96.25}\approx 9.8$ — about $2.4$ standard deviations up, a “significant” result. But look at where the evidence comes from: the three largest magnitudes, $9,12,15$, carry ranks $8,9,10$ and contribute $27$ of the $51$ on their own, while the small differences are split almost evenly in sign. Under a right-skewed population with median zero that pattern is typical, not evidence: the long right tail supplies large magnitudes far more often than the short left tail does, so the signs attached to the largest ranks are not fair coins, $E(W^{+})$ exceeds $\frac{n(n+1)}{4}$ under the null, and the test rejects a true null far more often than $\alpha$. The user would be “detecting” a shift whose existence the test had assumed away.

    Correction. With skewed differences, use the sign test: $S=8$ positives out of $10$, and under the null that the median difference is zero $S\sim \text{Binomial}(10,\frac{1}{2})$, with no further assumption — that null needs only $P(d>0)=\frac{1}{2}$. Then $P(S\ge 8)=\frac{\left(\genfrac{}{}{0ex}{}{10}{8}\right)+\left(\genfrac{}{}{0ex}{}{10}{9}\right)+\left(\genfrac{}{}{0ex}{}{10}{10}\right)}{1024}=\frac{45+10+1}{1024}=\frac{56}{1024}=\frac{7}{128}\approx 0.055$, two-sided about $0.11$: not significant, which is the honest reading of eight positives in ten.

    “Distribution-free” promises only that the null distribution of the statistic can be computed without knowing the population — for the sign test under any continuous distribution with median zero, for the signed-rank test under any distribution symmetric about zero. It does not mean assumption-free, and the assumption a rank test does make is exactly what fixes which null distribution one is entitled to use.

11. 11.Proof · 12 min

    Two independent samples have sizes $n_1$ and $n_2$, and the null hypothesis is that both come from the same continuous distribution. Let $N=\left(\genfrac{}{}{0ex}{}{n_1+n_2}{n_1}\right)$ and let $T$ be any statistic computed from the labelled data; assume $T$ takes $N$ distinct values across the $N$ relabellings. The permutation $p$-value is the proportion of relabellings with $T$ at least as large as the observed $T_{\text{obs}}$.

    1. (a)Prove that under the null, $P(p\le \alpha )=\lfloor \alpha N\rfloor /N$ for every $\alpha \in (0,1)$. Conclude that the test has level at most $\alpha$, with equality exactly when $\alpha N$ is an integer, and say precisely what “the permutation test is exact” means in the light of this. Point to the one place the null hypothesis is used.
    2. (b)Show that the smallest attainable one-sided $p$-value is $1/N$. For equal group sizes $n_1=n_2=m$, find the smallest $m$ for which a one-sided permutation test can reject at $\alpha =0.05$, and the smallest $m$ for which a two-sided test (with $p$ doubled) can.
    3. (c)Take $x=1,2,3$ and $y=4,5,6$ with $T=\bar{y}-\bar{x}$. Show that the one-sided $p$-value is exactly $0.05$, and then show that the same $p$-value is obtained if every observation is first replaced by its rank, or if $T$ is replaced by the Mann-Whitney $U$ of the second sample. What does this say about the relationship between rank tests and permutation tests?
    Show the solution to question 11

    (a) Exactness. Under the null the $n_1+n_2$ pooled observations are independent draws from one distribution, so their joint distribution is unchanged by any permutation of the labels: the observations are exchangeable. This is the only place the null hypothesis enters. Condition on the multiset of pooled values. Given that multiset, each of the $N$ ways of choosing which $n_1$ values carry the first label has the same conditional probability $1/N$, so the observed labelling is a uniform draw from the $N$ relabellings, and $T_{\text{obs}}$ is a uniform draw from the $N$ distinct values $t_{(1)}<t_{(2)}<\cdots <t_{(N)}$ that $T$ takes.

    If $T_{\text{obs}}=t_{(N-k+1)}$, the $k$-th largest value, then exactly $k$ relabellings have $T\ge T_{\text{obs}}$ and $p=k/N$. Hence $p$ is uniform on $\{\frac{1}{N},\frac{2}{N},\dots ,1\}$, and

    $$P(p\le \alpha )=P\left(k\le \alpha N\right)=\frac{\#\{k\in \{1,\dots ,N\}:k\le \alpha N\}}{N}=\frac{\lfloor \alpha N\rfloor }{N}\le \alpha .$$

    This is the conditional probability given the multiset; since it does not depend on the multiset, it is the unconditional probability too. Equality holds precisely when $\alpha N$ is an integer. $■$

    So “exact” means: the rejection probability under the null is known exactly, for every sample size, with no approximation — it is $\lfloor \alpha N\rfloor /N$, and it equals $\alpha$ whenever $\alpha$ is one of the attainable levels $k/N$. It does not mean that every $\alpha$ is attainable, and it says nothing about power. Nothing about the form of $T$ was used, which is why any statistic may be permuted.

    (b) The floor. The smallest $p$ occurs when $T_{\text{obs}}$ is the largest of the $N$ values, giving $p=1/N$; no relabelling can have $p$ smaller, since the observed one always counts. For $n_1=n_2=m$, $N=\left(\genfrac{}{}{0ex}{}{2m}{m}\right)$ runs $2,6,20,70,252,\dots$ for $m=1,2,3,4,5,\dots$

    One-sided at $\alpha =0.05$ requires $\frac{1}{N}\le 0.05$, i.e. $N\ge 20$: $m=2$ gives $N=6$ and fails, $m=3$ gives $N=20$ and succeeds, with $p=\frac{1}{20}=0.05$ exactly. So $m=3$ is the smallest, and only the single most extreme relabelling rejects.

    Two-sided with $p$ doubled requires $\frac{2}{N}\le 0.05$, i.e. $N\ge 40$: $m=3$ fails and $m=4$ gives $N=70$, $p=\frac{2}{70}=\frac{1}{35}\approx 0.029$. So $m=4$ is the smallest.

    (c) The example. The pooled values are $1,\dots ,6$ and $N=\left(\genfrac{}{}{0ex}{}{6}{3}\right)=20$. Since $\bar{y}-\bar{x}=\frac{(21-s)-s}{3}=\frac{21-2s}{3}$ where $s$ is the first group's sum, $T$ is largest when $s$ is smallest, and the observed labelling $\{1,2,3\}$ has the smallest possible sum $s=6$. It is the unique relabelling with $T\ge T_{\text{obs}}$, so $p=\frac{1}{20}=0.05$, exactly the floor from (b).

    Replacing values by ranks. The values $1,\dots ,6$ are their own ranks, so nothing changes and $p=\frac{1}{20}$. More generally, replacing observations by ranks is a strictly increasing transformation, and under it the ordering of relabellings by any statistic that is monotone in the group sum is preserved whenever, as here, the observed group holds the $n_1$ smallest values — that relabelling is the unique extreme one before and after ranking.

    Replacing $T$ by $U$. With $W$ the rank sum of the second sample, $U_y=W-\frac{n_2(n_2+1)}{2}=(4+5+6)-6=9$, which equals $n_1{n}_2=9$, the maximum possible: every $y$ exceeds every $x$. Across relabellings $U_y$ is a strictly increasing function of the second group's rank sum, hence a strictly decreasing function of $s$, so it orders the $20$ relabellings exactly as $T$ does, and again exactly one relabelling is at least as extreme: $p=\frac{1}{20}$.

    The lesson. The Mann-Whitney test is a permutation test — the permutation test whose statistic is the rank sum, applied to the ranks. Its tabulated null distribution is precisely the permutation distribution of $W$ over the $N$ relabellings, which is why it is the same for every continuous population: ranking has already removed the data's values, so the enumeration can be done once. A permutation test on the raw values keeps the magnitudes and can use any statistic, at the price of redoing the enumeration for each data set. Both are exact for the same reason, exchangeability under the null; they differ in what they throw away, and hence in power against particular alternatives.

12. 12.Computation · 10 min

    A sample of three values, $1,3,8$, is to be bootstrapped. Rather than simulate, enumerate: treat a resample as three independent draws, each uniform on $\{1,3,8\}$, and compute the bootstrap distribution exactly.

    1. (a)How many ordered resamples are there, and how many distinct resamples if order is ignored? What is the probability that a resample contains all three distinct values?
    2. (b)Find the exact bootstrap distribution of the resample median — the probability that it equals $1$, $3$ and $8$ — and check that the three probabilities sum to one.
    3. (c)Compute the bootstrap mean of the median and the bootstrap standard error of the median, both exactly.
    4. (d)Give the $95\%$ percentile interval for the median and explain what it reveals about bootstrapping a median from three observations. Then show, without enumerating, that the bootstrap standard error of the mean is exactly $\frac{\sqrt{26}}{3}$, and say why that statistic behaves better.
    Show the solution to question 12

    (a) Counting. Each of the three positions takes one of three values, so there are $3^3=27$ equally likely ordered resamples. Ignoring order, a resample is a multiset of size $3$ from $3$ types, of which there are $\left(\genfrac{}{}{0ex}{}{3+3-1}{3}\right)=\left(\genfrac{}{}{0ex}{}{5}{3}\right)=10$ — but these are not equally likely. A resample contains all three distinct values in $3!=6$ of the $27$ orderings, so the probability is $\frac{6}{27}=\frac{2}{9}$; the other $\frac{7}{9}$ of resamples repeat at least one observation, which is what makes sampling with replacement produce any variation at all.

    (b) The median. The resample median is the middle value of the three draws.

    The median is $1$ exactly when at least two of the three draws are $1$: two of them, $\left(\genfrac{}{}{0ex}{}{3}{2}\right){\left(\frac{1}{3}\right)}^2\frac{2}{3}=\frac{6}{27}$, or all three, ${\left(\frac{1}{3}\right)}^3=\frac{1}{27}$. So $P(\text{median}=1)=\frac{7}{27}$, and by the same count $P(\text{median}=8)=\frac{7}{27}$.

    The median is $3$ in the remaining cases — at least two $3$s ($\frac{7}{27}$ again) or all three values distinct ($\frac{6}{27}$, from (a)), which are disjoint — giving $P(\text{median}=3)=\frac{13}{27}$.

    Check: $\frac{7+13+7}{27}=1$. ✓

    (c) Moments of the bootstrap median. Writing $M^{\ast }$ for the resample median,

    $$E(M^{\ast })=\frac{7\cdot 1+13\cdot 3+7\cdot 8}{27}=\frac{7+39+56}{27}=\frac{102}{27}=\frac{34}{9},$$

    $$E\left(M^{\ast 2}\right)=\frac{7\cdot 1+13\cdot 9+7\cdot 64}{27}=\frac{7+117+448}{27}=\frac{572}{27}.$$

    Hence

    $$\mathrm{Var}(M^{\ast })=\frac{572}{27}-{\left(\frac{34}{9}\right)}^2=\frac{1716}{81}-\frac{1156}{81}=\frac{560}{81},$$

    and the bootstrap standard error of the median is

    $${\text{SE}}^{\ast }=\sqrt{\frac{560}{81}}=\frac{\sqrt{16\cdot 35}}{9}=\frac{4\sqrt{35}}{9}\approx \mathrm{2.63.}$$

    Note that the bootstrap mean $\frac{34}{9}\approx 3.78$ is above the observed median $3$: the resampling distribution has inherited the right skew of the data, since a resample that picks up the $8$ twice moves the median much further than one that picks up the $1$ twice.

    (d) The percentile interval. The bootstrap distribution has only three atoms. Its cumulative probabilities are $P(M^{\ast }\le 1)=\frac{7}{27}\approx 0.26$ and $P(M^{\ast }\le 3)=\frac{20}{27}\approx 0.74$. The $2.5\%$ quantile is the smallest value with cumulative probability at least $0.025$, which is $1$; the $97.5\%$ quantile is the smallest value with cumulative probability at least $0.975$, which is $8$. The $95\%$ percentile interval is $[1,8]$ — the entire range of the data. Indeed every percentile interval wider than the middle $48\%$ is $[1,8]$: the distribution is far too coarse to yield anything else. The bootstrap has not manufactured information three observations do not contain; a resample median is always one of the observed values, so with tiny $n$ the bootstrap distribution of the median is a handful of points and its quantiles are crude.

    The mean, without enumeration. A resample mean is the average of three independent draws from the empirical distribution, which puts mass $\frac{1}{3}$ on each of $1,3,8$. That distribution has mean $4$ and variance

    $$\frac{(1-4{)}^2+(3-4{)}^2+(8-4{)}^2}{3}=\frac{9+1+16}{3}=\frac{26}{3},$$

    so the variance of the average of three independent draws is $\frac{26}{3}\cdot \frac{1}{3}=\frac{26}{9}$ and

    $${\text{SE}}^{\ast }({\bar{x}}^{\ast })=\frac{\sqrt{26}}{3}\approx \mathrm{1.70.}$$

    This is just $\hat{\sigma }/\sqrt{n}$ with the divisor-$n$ standard deviation — the bootstrap reproduces the textbook formula because the mean is linear in the data. The mean behaves better as a bootstrap target because it varies continuously with the resample: the $27$ resamples give many distinct means spread smoothly over the range, whereas the median jumps between three values. The same reasoning is why a trimmed mean bootstraps more smoothly than a median at small $n$.