Design of Experiments

(a) Subtract the columns. ${\tau }_i=Y_i(1)-Y_i(0)$ gives $2,2,4,4,6,6$, so

$$\tau =\frac{2+2+4+4+6+6}{6}=\frac{24}{6}=4.$$

No experiment can compute a ${\tau }_i$ because each unit reveals only one column: once unit $i$ is treated it shows $Y_i(1)$ and $Y_i(0)$ becomes counterfactual, and if it is left untreated the reverse. Observing the unit more often, or more precisely, produces more copies of the same column. This is the fundamental problem of causal inference, and it is why the rest of the question works with the average $\tau$, which only needs each column to be observed on some units.

(b) Naive difference, then decompose. Units $4,5,6$ are treated and reveal $Y(1)=14,18,20$, mean $\frac{52}{3}$. Units $1,2,3$ are controls and reveal $Y(0)=4,6,8$, mean $6=\frac{18}{3}$. The naive difference is

$${\bar{Y}}_T-{\bar{Y}}_C=\frac{52}{3}-\frac{18}{3}=\frac{34}{3}\approx \mathrm{11.3.}$$

Effect on the treated: the mean of ${\tau }_i$ over units $4,5,6$ is $\frac{4+6+6}{3}=\frac{16}{3}$.

Selection bias: $E[Y(0)\mid T=1]-E[Y(0)\mid T=0]$ is the mean of $Y(0)$ over the treated, $\frac{10+12+14}{3}=12$, minus the mean over the controls, $6$; that is $6=\frac{18}{3}$.

Check: $\frac{16}{3}+\frac{18}{3}=\frac{34}{3}$. ✓

The naive figure is almost three times the truth $\tau =4$. Most of the gap is selection — the chosen units would have scored $6$ higher than the others under no treatment at all — and even the effect on the treated, $\frac{16}{3}$, exceeds $\tau$ because the strongest units also happen to have the largest ${\tau }_i$.

(c) Expectation without enumeration. Fix unit $i$. It is in the treated set exactly when the other two treated units are chosen from the remaining five, so it is treated in $\left(\genfrac{}{}{0ex}{}{5}{2}\right)=10$ of the $20$ assignments and $P(T_i=1)=\frac{1}{2}$ for every $i$. The treated mean is $\frac{1}{3}{\sum }_i{T}_i{Y}_i(1)$, and since the table is fixed and only $T$ is random,

$$E[{\bar{Y}}_T]=\frac{1}{3}\sum _{i=1}^6{Y}_i(1)P(T_i=1)=\frac{1}{6}\sum _i{Y}_i(1)=\frac{78}{6}=13,$$

using $6+8+12+14+18+20=78$. Identically $E[{\bar{Y}}_C]=\frac{1}{6}{\sum }_i{Y}_i(0)=\frac{54}{6}=9$. Hence $E[{\bar{Y}}_T-{\bar{Y}}_C]=13-9=4=\tau$. This is the theorem Randomisation identifies the average treatment effect in its smallest case: the random draw does not consult the table, so every unit's $Y(1)$ enters at the same weight $\frac{1}{2}$.

For the draw $\{1,3,5\}$: treated values $Y(1)=6,12,18$, mean $12$; controls $2,4,6$ with $Y(0)=6,10,14$, mean $10$; naive difference $2$. Decomposing as in (b): effect on the treated $\frac{2+4+6}{3}=4$, selection bias $\frac{4+8+12}{3}-\frac{6+10+14}{3}=8-10=-2$, and $4-2=2$. ✓

What this means: a single randomised draw is not free of selection bias — here it is $-2$, and the estimate $2$ is half the truth. Randomisation promises that the bias is zero on average across draws, which is what “unbiased” says, and the spread of that bias term from draw to draw is exactly what a standard error measures. What it does not mean is that this experiment went wrong: nothing about the draw could have been checked against a truth the experimenter never sees.

(a) Sentence 2 — re-randomising until the table looks balanced. The theorem Randomisation identifies the average treatment effect needs $T\perp (Y(0),Y(1))$. Discarding every allocation in which age is unbalanced makes the final assignment a function of the covariates: allocations that put older volunteers in one arm are systematically thrown away, so the mechanism is no longer independent of anything correlated with age — and pain is. The imbalance was expected, not a failure: testing fifteen baseline variables at $5\%$ gives a probability $1-0.95^{15}$ of at least one “significant” difference by chance alone, which is roughly even odds (about $54\%$). Correct procedure: keep the first allocation. If age is known to matter, block on it in advance — randomise within age strata — or pre-specify an adjustment for age in the analysis.

Sentence 3 — “identical”. Randomisation never promises identical groups; with forty per arm they will visibly differ on something. It promises that any difference is accidental, hence governed by known probability, hence exactly what the standard error measures. “No significant difference” is a statement about a test's power, not about equality. And “any later difference must be the drug” ignores everything that happens after assignment, which is the next sentence's problem.

Sentence 4 — unblinded assessment, and no placebo. Randomisation makes the groups comparable before treatment; blinding keeps them comparable afterwards. Pain is reported by the patient and scored by the clinician, so a patient who knows they received the drug reports differently and a clinician who knows the assignment scores differently, and the comparison is contaminated even though the randomisation was sound. Correct procedure: the no-drug arm receives a placebo indistinguishable from the drug (single-blind), and the assessors do not know assignments (double-blind).

Sentence 5 — extrapolation. The population randomised was volunteers aged $20$ to $35$. Randomisation buys internal validity — the comparison inside the study is sound — and buys external validity not at all; nothing in the trial speaks to the elderly, whose physiology, dosing and baseline pain differ. Correct procedure: state the conclusion for the population studied, and run a trial in the elderly if they are the population of interest.

(b) The strongest defensible claim. With one allocation kept as drawn, a placebo arm, double-blind assessment, and the analysis pre-specified: the difference in mean pain score between arms is an unbiased estimate of the average effect of the drug relative to placebo, in volunteers aged $20$ to $35$ of the kind who enrolled, with a standard error that the randomisation licenses. That is a causal claim about that population — and about no other.

(a) Independent plots, so variances add. Every plot is on its own terrace, so all twenty yields are independent. Each arm's mean is a mean of ten independent plots with variance ${\sigma }^2/10$, and the two arm means are independent of each other, so

$$\mathrm{Var}({\bar{Y}}_T-{\bar{Y}}_C)=\frac{36}{10}+\frac{36}{10}=\frac{72}{10}=\frac{36}{5}=\mathrm{7.2.}$$

(b) Difference within a terrace. For terrace $i$ let $D_i=Y_{iT}-Y_{iC}$. Then

$$\mathrm{Var}(D_i)={\sigma }^2+{\sigma }^2-2\rho {\sigma }^2=36+36-2\cdot \frac{1}{2}\cdot 36=36=2{\sigma }^2(1-\rho ).$$

The ten terraces are independent, so $\mathrm{Var}(\bar{D})=\frac{36}{10}=\frac{18}{5}=3.6$. The ratio is $\frac{3.6}{7.2}=\frac{1}{2}=1-\rho$, as the theorem Variance reduction from blocking predicts.

(c) Matching precision, and a fourfold gain. With $m$ plots per schedule Design U has variance $2{\sigma }^2/m=72/m$. Setting $72/m=3.6$ gives $m=20$ per schedule, so $40$ plots — twice Design B's twenty. For a fourfold reduction we need $1-\rho =\frac{1}{4}$, that is $\rho =\frac{3}{4}$.

(d) Blocking on nothing. With $\rho =0$, $\mathrm{Var}(\bar{D})=2{\sigma }^2/10=7.2$, the same as Design U. But the analyses differ in degrees of freedom: Design U is a two-sample comparison of $10$ against $10$, with $20-2=18$ error degrees of freedom, while Design B is a paired comparison of ten differences with $10-1=9$. The $t$ critical value on $9$ degrees of freedom is larger than on $18$, so Design B is slightly less powerful for the same variance: the degrees of freedom spent on terrace effects bought nothing. Design U is now the better design. Blocking is only worth doing on a variable genuinely related to the outcome — and whichever design is run, it must be analysed as designed: paired data analysed as two independent samples is invalid, not merely inefficient.

(a) One square, and the property that matters. Index rows by day and columns by car:

$$\begin{array}{cccc}& \text{Car }1& \text{Car }2& \text{Car }3\\ \text{Day }1& A& B& C\\ \text{Day }2& B& C& A\\ \text{Day }3& C& A& B\end{array}$$

Each compound appears exactly once in every row and exactly once in every column. So the three runs of any one compound cover each day once and each car once, and a day effect (or a car effect) adds the same total to every compound. It therefore cancels in every difference between compounds, which is what “cannot be confounded” means. Any other cyclic shift of the rows works equally well.

(b) Degrees of freedom. Nine runs give $9$ degrees of freedom in total: $1$ for the mean, $2$ for cars, $2$ for days, $2$ for compounds, leaving

$$9-1-2-2-2=2$$

for error. A full factorial crossing three factors at three levels needs $3\times 3\times 3=27$ runs, three times the budget.

(c) The proposal confounds compound with car. The colleague's grid has every row $ABC$. Rows are fine — each compound once per day — but the columns fail: $A$ appears three times in the car 1 column and never elsewhere. Compound $A$ is only ever observed on car 1, $B$ only on car 2, $C$ only on car 3, so the compound-to-compound difference $A-B$ is inseparable from the car-to-car difference $1-2$. Any car effect looks like a compound effect and nothing in the nine numbers can tell them apart. The layout controls one nuisance (days) and aliases the compound comparison with the other. That is a one-way blocking, not a Latin square.

(d) Additivity, and why it cannot be checked. The Latin square assumes there are no interactions among day, car and compound — that each acts additively. All $9-1=8$ non-mean degrees of freedom are spent on the three sets of main effects and the $2$ error degrees of freedom; there is no run left over to estimate any interaction, so a car-by-compound interaction, if present, is silently absorbed into the error and the compound estimates. The $27$-run factorial has $26-6=20$ degrees of freedom for interactions and can test them; the square cannot. It buys its efficiency by assuming what the factorial would have measured.

(a) Effects from the cell means. $A$'s effect at $B$ low is $50-40=10$ and at $B$ high is $50-46=4$, so the main effect of $A$ is the average, $\frac{10+4}{2}=7$. $B$'s effect at $A$ low is $46-40=6$ and at $A$ high is $50-50=0$, so the main effect of $B$ is $\frac{6+0}{2}=3$.

Interaction from $A$'s two effects: $\frac{4-10}{2}=-3$. From $B$'s two effects: $\frac{0-6}{2}=-3$. ✓ Both are the same contrast $\frac{1}{2}({\bar{y}}_{11}-{\bar{y}}_{10}-{\bar{y}}_{01}+{\bar{y}}_{00})=\frac{1}{2}(50-50-46+40)=-3$, which is why they had to agree.

(b) One standard error for all three. Each estimate is a contrast in the four independent cell means with coefficients $\pm \frac{1}{2}$, so

$$\mathrm{Var}=\sum c_i^2\cdot \frac{MSE}{n}=4\cdot \frac{1}{4}\cdot \frac{16}{4}=4,SE=2.$$

Error degrees of freedom: four cells each contribute $n-1=3$, so $12$. The statistics are $t_A=\frac{7}{2}=3.5$, $t_B=\frac{3}{2}=1.5$, $t_{AB}=\frac{-3}{2}=-1.5$. Against $2.179$: temperature is significant; airflow and the interaction are not.

(c) Reporting. Because the interaction is not significant, the main effect of $A$ is a legitimate one-number summary: the data do not establish that $A$'s effect depends on $B$. But the point estimates do differ — $10$ at low airflow, $4$ at high — and the test had little power to see it (an estimate of $-3$ against a standard error of $2$). Not significant is not the same as zero. Word it honestly: “Raising the temperature increased drying time by $7$ on average (SE $2$). The increase appeared larger at low airflow ($10$ against $4$), but the difference was not significant ($t=-1.5$ on $12$ degrees of freedom).”

(d) The one-factor-at-a-time study. Its estimate is ${\bar{y}}_{10}-{\bar{y}}_{00}$ with $8$ runs at each condition, so

$$\mathrm{Var}=\frac{16}{8}+\frac{16}{8}=4,$$

exactly the factorial's. That is the theorem Factorial designs estimate more from the same runs with $N=16$: both designs give $4{\sigma }^2/N=64/16=4$ for $A$. The difference is what else is learned. The one-factor study estimates $10$, valid only at low airflow; it cannot estimate $B$'s effect, cannot estimate the interaction, and cannot discover that the $10$ shrinks to $4$ when airflow is high. The factorial got $A$ at the same precision and $B$ and $AB$ for free.

(a) The three effects. Experiment I: $A=\frac{(16-10)+(20-14)}{2}=\frac{6+6}{2}=6$, $B=\frac{(14-10)+(20-16)}{2}=\frac{4+4}{2}=4$, $AB=\frac{6-6}{2}=0$.

Experiment II: $A=\frac{(16-10)+(8-14)}{2}=\frac{6-6}{2}=0$, $B=\frac{(14-10)+(8-16)}{2}=\frac{4-8}{2}=-2$, $AB=\frac{-6-6}{2}=-6$.

(b) The two plots. Experiment I: the $B$-low segment runs from $(\text{low},10)$ to $(\text{high},16)$ and the $B$-high segment from $(\text{low},14)$ to $(\text{high},20)$. Both rise by $6$; they are parallel, with a vertical gap of $4$ at each end. Experiment II: the $B$-low segment again runs from $10$ up to $16$, but the $B$-high segment runs from $14$ down to $8$. The picture is an X.

What each feature shows. The slope of a segment is $A$'s effect at that level of $B$, so the main effect of $A$ is the average slope: $6$ in I; $0$ in II, where one segment rises by exactly what the other falls. The main effect of $B$ is the average vertical distance from the $B$-low segment up to the $B$-high segment: $4$ in I; in II it is $+4$ at the left end and $-8$ at the right, averaging $-2$. The interaction is half the difference of the slopes: $\frac{6-6}{2}=0$ and $\frac{-6-6}{2}=-6$. Zero interaction therefore looks like parallel segments, as in I: $A$ does the same thing whatever $B$ is. Crossing segments, as in II, mean the interaction is so large that $A$'s effect reverses sign.

(c) The crossing, and the one-factor report. Let $t\in [0,1]$ measure the way across from $A$ low to $A$ high. The $B$-low segment has height $10+6t$ and the $B$-high segment $14-6t$; they meet when $12t=4$, so $t=\frac{1}{3}$, at height $10+2=12$. The segments cross one third of the way across, at $12$. (The horizontal axis is categorical, so this is a feature of the drawing, not a prediction about an intermediate temperature.)

An experimenter who tests $A$ at $B$ low sees ${\bar{y}}_{10}-{\bar{y}}_{00}=16-10=6$ in both experiments and reports “raising $A$ adds $6$” for both. In Experiment I that is the whole truth. In Experiment II it is true only at $B$ low; at $B$ high, raising $A$ subtracts $6$, and the one-factor design never ran the cell that would have shown it. The factorial's main effect of $A$ is $0$, which quoted alone would be equally misleading — “$A$ does nothing” is false at both levels of $B$. The honest report is conditional: $A$ adds $6$ at low $B$ and removes $6$ at high $B$, the interaction is $-6$, and no single number for $A$'s effect exists.

(a) An identity that holds for every design. On the event $T=1$ the observed outcome is $Y=Y(1)$, and on $T=0$ it is $Y=Y(0)$. Both events have positive probability, so the conditional expectations exist and

$$E[Y\mid T=1]=E[Y(1)\mid T=1],E[Y\mid T=0]=E[Y(0)\mid T=0].$$

Now add and subtract $E[Y(0)\mid T=1]$:

$$E[Y(1)\mid T=1]-E[Y(0)\mid T=0]=(E[Y(1)\mid T=1]-E[Y(0)\mid T=1])+(E[Y(0)\mid T=1]-E[Y(0)\mid T=0]).$$

By linearity of conditional expectation the first bracket is $E[Y(1)-Y(0)\mid T=1]={\tau }_{\mathrm{ATT}}$, which is the identity. $■$

In words: ${\tau }_{\mathrm{ATT}}$ is the average effect of the treatment on the units that were actually treated. The second bracket is selection bias — how the treated units would have differed from the controls had nobody been treated, a pure baseline difference. Nothing was assumed, so every design faces this identity; a design can only hope to make the second bracket vanish.

(b) Randomisation kills the bracket. Independence says the conditional distribution of $(Y(0),Y(1))$ given $T$ is its unconditional distribution. Therefore

$$E[Y(0)\mid T=1]=E[Y(0)]=E[Y(0)\mid T=0]\text{(independence used here, and only here)},$$

so the bracket is $0$. By the same fact $E[Y(1)\mid T=1]=E[Y(1)]$ and $E[Y(0)\mid T=1]=E[Y(0)]$, so

$${\tau }_{\mathrm{ATT}}=E[Y(1)]-E[Y(0)]=E[Y(1)-Y(0)]=\tau .$$

Putting these into (a): $E[Y\mid T=1]-E[Y\mid T=0]=\tau$. The group mean ${\bar{Y}}_T$ averages $Y_i(1)$ over the treated units, and for each of them $E[Y_i(1)\mid T]=E[Y_i(1)]$ by independence, so $E[{\bar{Y}}_T]=E[Y(1)]$ whenever the treated group is non-empty; likewise $E[{\bar{Y}}_C]=E[Y(0)]$. Hence $E[{\bar{Y}}_T-{\bar{Y}}_C]=\tau$. $■$

This is the theorem Randomisation identifies the average treatment effect, reached through (a) so that one can see what randomisation is removing: a selection term that is otherwise present and invisible.

(c) A biased coin is still a coin. The conclusion survives. The proof used one hypothesis — $T\perp (Y(0),Y(1))$ — and a coin with bias $p$ satisfies it for every $p$, because the coin does not consult the potential outcomes. The only place $p$ could have entered is the marked step: the claim $E[Y(0)\mid T=1]=E[Y(0)]$ says the conditional distribution given $T=1$ equals the marginal, and that is true whatever $P(T=1)$ is, provided it is positive so that the conditioning makes sense. The same holds for the $T=0$ side, which is why $p<1$ is needed too.

What $p$ does affect is precision, not bias. The group sizes are about $Np$ and $N(1-p)$, so the variance of ${\bar{Y}}_T-{\bar{Y}}_C$ is about $\frac{{\sigma }_1^2}{Np}+\frac{{\sigma }_0^2}{N(1-p)}$, which for equal variances is smallest at $p=\frac{1}{2}$ and grows without bound as $p$ approaches $0$ or $1$. Unbiasedness is a property of independence; efficiency is a property of balance.

(a) Unpaired. $z_{\alpha /2}+z_{\beta }=1.96+0.84=2.80$, so $(z_{\alpha /2}+z_{\beta }{)}^2=7.84$. The theorem gives, per coating,

$$n\ge \frac{2(7.84)(100)}{25}=\frac{1568}{25}=62.72,$$

so $63$ replicates per coating and $126$ panels in total. (Check the arithmetic: $2\times 7.84=15.68$, times $100$ is $1568$, and $25\times 62=1550$ with remainder $18$, so $62.72$.)

(b) Paired. Blocking multiplies the error variance by $1-\rho$, so the effective ${\sigma }^2$ is $100\times \frac{1}{4}=25$, and

$$n\ge \frac{2(7.84)(25)}{25}=15.68,$$

so $16$ pairs, $32$ panels. The bounds before rounding are in the ratio $\frac{15.68}{62.72}=\frac{1}{4}=1-\rho$ exactly; after rounding, $\frac{32}{126}=\frac{16}{63}$, which is $\frac{1}{4}$ up to the rounding to whole panels. The pairing has cut the experiment to a quarter of its size, which is the factor $(1-\rho )$ applied to the variance.

(c) Halving the effect. $\delta$ enters the bound only through ${\delta }^2$ in the denominator, so halving $\delta$ multiplies the bound by $4$: unpaired $4\times 62.72=250.88$, so $251$ per coating and $502$ panels; paired $4\times 15.68=62.72$, so $63$ pairs and $126$ panels. Round the bound, not the rounded $n$ — $4\times 63=252$ would be one panel too many. The standardised effect has gone from $\delta /\sigma =\frac{1}{2}$ to $\frac{1}{4}$, and small effects are expensive.

(d) What must be fixed in advance. Everything on the right-hand side: $\alpha$, the power $1-\beta$, the effect $\delta$ worth detecting, and $\sigma$. None of them comes from the data being collected. The suggestion goes wrong in two ways. First, an estimate of $\sigma$ from ten panels is itself very noisy — its own sampling error is large — so the chosen $n$ will be badly wrong in one direction or the other, and a design sized by a lucky low estimate will be underpowered. Second, if those ten panels are also part of the final analysis, then the sample size — and so the analysis — was chosen after looking at outcomes, and the test's stated level is no longer guaranteed. The value of $\sigma$ should come from earlier work or from a separate pilot that is not part of the experiment being analysed, and $\delta$ from a judgement about what matters, settled before the first panel is cut.

(a) Six rates. New drug: mild $\frac{40}{50}=80\%$, severe $\frac{60}{150}=40\%$, overall $\frac{100}{200}=50\%$. Standard drug: mild $\frac{105}{150}=70\%$, severe $\frac{15}{50}=30\%$, overall $\frac{120}{200}=60\%$. The totals are consistent: $40+60=100$ of $50+150=200$, and $105+15=120$ of $150+50=200$.

The new drug is better in mild cases ($80$ against $70$) and better in severe cases ($40$ against $30$), by ten points each time — and worse overall ($50$ against $60$). That is Simpson's paradox.

(b) Severity is the confounder. It is associated with treatment: the new drug went to severe cases $150$ times out of $200$ ($75\%$), while the standard drug went to mild cases $150$ times out of $200$ ($75\%$). It is associated with outcome under either drug: severe cases succeed at $40\%$ or $30\%$ against $80\%$ or $70\%$ for mild. And it is not on the causal pathway — severity was assessed at presentation, before any drug was given.

The overall rates are therefore weighted averages with different weights. New drug: $\frac{1}{4}(80)+\frac{3}{4}(40)=20+30=50$. Standard: $\frac{3}{4}(70)+\frac{1}{4}(30)=52.5+7.5=60$. The new drug's overall figure is dragged down by carrying three quarters of its patients in the hard stratum. The reversal is the case mix, not the drug.

(c) Adjust by stratification. Stratum sizes over both drugs: mild $50+150=200$, severe $150+50=200$, so both weights are $\frac{1}{2}$.

$$\text{New: }\frac{1}{2}(0.8)+\frac{1}{2}(0.4)=0.6,\text{Standard: }\frac{1}{2}(0.7)+\frac{1}{2}(0.3)=\mathrm{0.5.}$$

The adjusted difference is $0.6-0.5=+0.10$: the new drug is ten points better, agreeing with both strata. The unadjusted difference was $0.50-0.60=-0.10$. Adjustment has moved the estimate by $0.20$ and flipped its sign; that $0.20$ is the size of the confounding by severity.

(d) The assumption, and a variable that must not be adjusted for. The adjusted $0.10$ equals the average treatment effect if $T\perp (Y(0),Y(1))\mid \text{severity}$ — no unmeasured confounding: within a severity stratum, which drug a patient got is unrelated to how they would have fared under either. That is the hypothesis of the theorem Adjustment identifies the effect only under no unmeasured confounding, and the table cannot check it. If, say, doctors also gave the new drug to the sicker patients within the severe stratum, the adjusted figure still understates the new drug, and nothing in these numbers reveals it.

A variable for which the arithmetic would be an error: “fever resolved within $48$ hours of starting treatment”. That is a consequence of the drug, on the pathway from drug to cure. Stratifying on it compares patients whose fever did resolve with each other, and those whose fever did not with each other, which removes the part of the drug's effect that operates through resolving fever; the adjusted difference would be pushed toward zero and could vanish for a drug that works. Whether to condition on a variable is a causal question — a pre-treatment confounder yes, a post-treatment consequence no — and the weighting arithmetic is the same either way, so it cannot decide.

(a) The errors, step by step.

Step 1 — adjusting for post-treatment variables. Two of the forty are consequences of the treatment. Earnings in the year after the programme lie on the causal pathway from training to earnings three years later: a mediator. Conditioning on it compares participants and non-participants who had the same early earnings, which removes the part of training's effect that works through an early earnings gain, and biases the estimate toward zero. Promotion in the following year is caused by training and by the earning-related ability that also drives later earnings: a variable caused by both treatment and outcome is a collider, and conditioning on it manufactures an association between training and earnings that does not exist, in a direction the memo cannot know. Both must be dropped. An adjustment set is a set of pre-treatment confounders, chosen from knowledge of the causal structure before the analysis.

Step 2 — “verified”. Balance on the measured variables is produced by the adjustment itself; that is what adjusting does, so it is not evidence of anything. The assumption $T\perp (Y(0),Y(1))\mid X$ concerns the variables not in $X$, and it is untestable from the data precisely because those variables were not recorded. No balance table can verify it.

Step 3 — “therefore”. The causal reading follows only under the assumption of Step 2, which was assumed, not established. With voluntary enrolment the obvious unmeasured confounder is motivation, which raises both the chance of enrolling and later earnings, whatever the training does; the adjusted difference then contains a selection term of unknown size.

Step 4 — “in the same sense”. A randomised trial earns $T\perp (Y(0),Y(1))$ by construction; the theorem Adjustment identifies the effect only under no unmeasured confounding takes the conditional version as a hypothesis. One estimate is unbiased because of the design, the other only if an unverifiable assumption happens to hold. The theorem is true; the memo has used it as if its hypothesis were free.

Step 5 — “can only reduce bias”. Step 1 already shows why this is false: adding a mediator removes part of the effect and adding a collider creates a spurious one. More adjustment is not uniformly safer; which variables to adjust for is a causal question, not a statistical one, and “everything available” is not a safe default.

(b) A corrected conclusion. Dropping the two post-programme variables and adjusting for the pre-enrolment covariates — age, schooling, prior earnings, region and the other variables measured before enrolment — the adjusted difference estimates the average effect of training on earnings three years later under the assumption that, among people with the same values of those covariates, the decision to enrol is independent of what they would have earned with and without training. This assumption cannot be checked from the data. It fails if, among people alike on every recorded variable, those who enrolled were systematically more likely to earn more anyway (the more motivated, the better connected) or systematically less likely (the recently laid off, enrolling out of need); either direction is possible, and the sign of the resulting bias is unknown.

(c) A design that changes the assumption. If the programme was oversubscribed and places were allocated by lottery among applicants, the lottery is a natural experiment: within the applicant pool, assignment is independent of the potential outcomes by construction, and the comparison of winners against losers is a randomised comparison. The assumption it relies on is that the lottery really was random and its outcome is what determined participation (with non-compliance, the comparison by lottery result remains valid as an intention-to-treat estimate). Alternatively, if eligibility was decided by a threshold — prior earnings below a cutoff — a regression discontinuity compares people just either side of it, under the assumption that they are comparable in everything except eligibility. Each replaces “no unmeasured confounding” with a different assumption, not with none; the gain is that these assumptions can be argued from how the allocation actually worked, rather than hoped for.

(a) Averaging within a chamber. Averaging the model over $j=1,\dots ,m$,

$${\bar{Y}}_{i\cdot }=\mu +{\tau }_{g(i)}+C_i+{\bar{e}}_{i\cdot },{\bar{e}}_{i\cdot }=\frac{1}{m}\sum _{j=1}^m{e}_{ij}.$$

The $e_{ij}$ are independent with common variance, so $\mathrm{Var}({\bar{e}}_{i\cdot })=\frac{1}{m^2}\cdot m{\sigma }_e^2=\frac{{\sigma }_e^2}{m}$. The constants have no variance, and $C_i$ is independent of ${\bar{e}}_{i\cdot }$, so the variances add: $\mathrm{Var}({\bar{Y}}_{i\cdot })={\sigma }_c^2+{\sigma }_e^2/m$.

For two plants $j\ne j^{\prime }$ in the same chamber, $\mathrm{Cov}(Y_{ij},Y_{i{j}^{\prime }})=\mathrm{Cov}(C_i+e_{ij},C_i+e_{i{j}^{\prime }})=\mathrm{Var}(C_i)={\sigma }_c^2$, since every cross term involves two independent mean-zero variables and vanishes. Each plant has variance ${\sigma }_c^2+{\sigma }_e^2={\sigma }^2$, so the correlation is ${\sigma }_c^2/{\sigma }^2=\rho$. $■$

(b) The true variance and the analyst's. Because every chamber holds the same number $m$ of plants, the mean over all $km$ plants at one setting equals the mean of its $k$ chamber means: ${\bar{Y}}_T=\frac{1}{k}{\sum }_{i\in T}{\bar{Y}}_{i\cdot }$. Different chambers share no $C$ and no $e$, so the chamber means are independent, and by (a)

$$\mathrm{Var}({\bar{Y}}_T)=\frac{1}{k}({\sigma }_c^2+\frac{{\sigma }_e^2}{m}).$$

The two settings use disjoint chambers, so ${\bar{Y}}_T$ and ${\bar{Y}}_C$ are independent and $\mathrm{Var}({\bar{Y}}_T-{\bar{Y}}_C)=\frac{2}{k}({\sigma }_c^2+{\sigma }_e^2/m)$.

The analyst who takes the plants as $km$ independent units of variance ${\sigma }^2$ computes $\mathrm{Var}({\bar{Y}}_T)={\sigma }^2/(km)$ and hence $2{\sigma }^2/(km)$ for the difference. $■$ That is pseudoreplication: the treatment was applied to chambers, so the experimental unit is the chamber and there are $k$ of them per setting, not $km$.

(c) The ratio. Divide:

$$\frac{\frac{2}{k}({\sigma }_c^2+{\sigma }_e^2/m)}{2{\sigma }^2/(km)}=\frac{m{\sigma }_c^2+{\sigma }_e^2}{{\sigma }^2}=\frac{m{\sigma }_c^2+{\sigma }^2-{\sigma }_c^2}{{\sigma }^2}=1+(m-1)\frac{{\sigma }_c^2}{{\sigma }^2}=1+(m-1)\rho .$$

Since $0\le \rho \le 1$ the ratio of variances lies in $[1,m]$, so the ratio of standard errors — true to reported — lies in $[1,\sqrt{m}]$. $■$

The lower extreme $1$ is attained exactly when $\rho =0$, that is ${\sigma }_c^2=0$: no chamber effect, so plants really are independent units and the analyst was right. The upper extreme $\sqrt{m}$ is attained exactly when $\rho =1$, that is ${\sigma }_e^2=0$: plants in a chamber are identical, so $m$ measurements carry one unit's worth of information. The rule of thumb “too small by about $\sqrt{m}$” is the worst case; the exact factor is $\sqrt{1+(m-1)\rho }$.

(d) Numbers, the limit, and the budget. Here ${\sigma }^2=3$ and $\rho =\frac{1}{3}$. True variance: $\frac{2}{3}(1+\frac{2}{10})=\frac{2}{3}\cdot \frac{6}{5}=\frac{12}{15}=\frac{4}{5}$. Analyst's: $\frac{2\cdot 3}{30}=\frac{1}{5}$. Ratio $4$, and $1+9\cdot \frac{1}{3}=4$ agrees. ✓ The reported standard error is too small by a factor of $2$: a reported $t=5$ is really $t=2.5$, and on $2k-2=4$ degrees of freedom rather than $58$.

As $m\to \infty$ with $k$ fixed, ${\sigma }_e^2/m\to 0$ and the true variance tends to $2{\sigma }_c^2/k=\frac{2}{3}$, not to zero. More plants average away the plant error but never the chamber effect, which is shared by every plant in the chamber.

Thirty extra plants per setting. Three more chambers of ten: $k=6$, $m=10$, variance $\frac{2}{6}\cdot \frac{6}{5}=\frac{2}{5}=0.4$ — halved. Ten more plants in each existing chamber: $k=3$, $m=20$, variance $\frac{2}{3}(1+\frac{2}{20})=\frac{2}{3}\cdot \frac{11}{10}=\frac{11}{15}\approx 0.73$ — down from $\frac{12}{15}$ by one part in twelve, and no number of extra plants could take it below $\frac{2}{3}$. The chambers win by a wide margin. Replication means more experimental units, and the experimental unit is whatever the treatment was applied to.

(a) Coefficient vectors. Expanding the brackets,

$$A=\frac{1}{2}(-{\bar{y}}_{00}+{\bar{y}}_{10}-{\bar{y}}_{01}+{\bar{y}}_{11}),B=\frac{1}{2}(-{\bar{y}}_{00}-{\bar{y}}_{10}+{\bar{y}}_{01}+{\bar{y}}_{11}),AB=\frac{1}{2}({\bar{y}}_{00}-{\bar{y}}_{10}-{\bar{y}}_{01}+{\bar{y}}_{11}),$$

so in the order $(00,10,01,11)$ the vectors are

$$c_A=(-\frac{1}{2},\frac{1}{2},-\frac{1}{2},\frac{1}{2}),c_B=(-\frac{1}{2},-\frac{1}{2},\frac{1}{2},\frac{1}{2}),c_{AB}=(\frac{1}{2},-\frac{1}{2},-\frac{1}{2},\frac{1}{2}).$$

Each has two entries $+\frac{1}{2}$ and two $-\frac{1}{2}$, so each sums to zero. Pattern: $c_A$ is $+$ where $A$ is high, $c_B$ is $+$ where $B$ is high, and $c_{AB}$ is the entrywise product of the two. Summing to zero makes each a contrast: adding the same constant to all four cells changes none of them, which is why the overall level never enters an effect.

(b) Variance. The cell means are independent, so for any coefficient vector

$$\mathrm{Var}\left(\sum _j{c}_j{\bar{y}}_j\right)=\sum _j{c}_j^2\mathrm{Var}({\bar{y}}_j)=\frac{{\sigma }^2}{n}\sum _j{c}_j^2.$$

For each of the three vectors ${\sum }_j{c}_j^2=4\cdot \frac{1}{4}=1$, so the variance is ${\sigma }^2/n$. With $N=4n$ this is ${\sigma }^2/(N/4)=4{\sigma }^2/N$. $■$ A one-factor-at-a-time study with $N/2$ runs at each of two conditions estimates its one effect as a difference of two means, with variance $\frac{{\sigma }^2}{N/2}+\frac{{\sigma }^2}{N/2}=\frac{4{\sigma }^2}{N}$ — the same. This is the theorem Factorial designs estimate more from the same runs: the factorial reaches that precision for all three effects at once.

(c) Orthogonality. For two coefficient vectors $c$ and $d$, bilinearity of covariance and independence of the cells give

$$\mathrm{Cov}(\sum _j{c}_j{\bar{y}}_j,\sum _l{d}_l{\bar{y}}_l)=\sum _j\sum _l{c}_j{d}_l\mathrm{Cov}({\bar{y}}_j,{\bar{y}}_l)=\frac{{\sigma }^2}{n}\sum _j{c}_j{d}_j,$$

since every cross-cell covariance is zero. The three inner products, in the order $(00,10,01,11)$:

$$c_A\cdot c_B=\frac{1}{4}-\frac{1}{4}-\frac{1}{4}+\frac{1}{4}=0,c_A\cdot c_{AB}=-\frac{1}{4}-\frac{1}{4}+\frac{1}{4}+\frac{1}{4}=0,c_B\cdot c_{AB}=-\frac{1}{4}+\frac{1}{4}-\frac{1}{4}+\frac{1}{4}=0.$$

All three covariances vanish. $■$

This is hidden replication made precise. No coefficient in any vector is zero, so every run enters every estimate — nothing is set aside for $B$ that is then unavailable for $A$. And yet the estimates do not compete for information: they are orthogonal, so estimating $B$ costs $A$ nothing in precision, and estimating $AB$ costs neither. Each effect is estimated exactly as well as if it were the only thing the experiment was for.

(d) Recovering the cells. Write out $\bar{\mu }+\frac{1}{2}A+\frac{1}{2}B+\frac{1}{2}AB$ and collect the coefficient of each cell mean; every term contributes $\pm \frac{1}{4}$.

${\bar{y}}_{11}$: $\frac{1}{4}+\frac{1}{4}+\frac{1}{4}+\frac{1}{4}=1$. ${\bar{y}}_{00}$: $\frac{1}{4}-\frac{1}{4}-\frac{1}{4}+\frac{1}{4}=0$. ${\bar{y}}_{10}$: $\frac{1}{4}+\frac{1}{4}-\frac{1}{4}-\frac{1}{4}=0$. ${\bar{y}}_{01}$: $\frac{1}{4}-\frac{1}{4}+\frac{1}{4}-\frac{1}{4}=0$.

So the expression equals ${\bar{y}}_{11}$. $■$ The sign rule: give each factor the sign $s=+1$ at its high level and $s=-1$ at its low level; then

$${\bar{y}}_{ab}=\bar{\mu }+\frac{1}{2}{s}_{a}A+\frac{1}{2}{s}_{b}B+\frac{1}{2}{s}_a{s}_{b}AB,$$

the interaction carrying the product of the signs. Explicitly ${\bar{y}}_{10}=\bar{\mu }+\frac{1}{2}A-\frac{1}{2}B-\frac{1}{2}AB$, ${\bar{y}}_{01}=\bar{\mu }-\frac{1}{2}A+\frac{1}{2}B-\frac{1}{2}AB$, and ${\bar{y}}_{00}=\bar{\mu }-\frac{1}{2}A-\frac{1}{2}B+\frac{1}{2}AB$; each checks by the same coefficient count (for ${\bar{y}}_{00}$ the four contributions to its own coefficient are $\frac{1}{4}+\frac{1}{4}+\frac{1}{4}+\frac{1}{4}$, and every other cell's cancel in pairs).

Sanity check with cell means $40,50,46,50$: $\bar{\mu }=46.5$, $A=7$, $B=3$, $AB=-3$, and ${\bar{y}}_{11}=46.5+3.5+1.5-1.5=50$, ${\bar{y}}_{00}=46.5-3.5-1.5-1.5=40$. ✓

Why exactly three effects: the map from the four cell means to $(\bar{\mu },A,B,AB)$ is invertible — part (d) is its inverse — so the four numbers on each side carry the same information. The four coefficient vectors, $(\frac{1}{4},\frac{1}{4},\frac{1}{4},\frac{1}{4})$ and the three contrasts, are mutually orthogonal and so form a basis of the four-dimensional space of cell-mean combinations. One dimension is the overall level; the remaining $2^2-1=3$ are the effects. Any further contrast is a linear combination of $c_A$, $c_B$ and $c_{AB}$, so a “fourth effect” would be a relabelling of these three, not new information. This is the proposition How many effects a two-level factorial estimates in the case $k=2$, with the dimension count made explicit.