Statistics · Solutions

Statistical Inference: Estimation

12 questions · about 108 minutes

Work this sheet on paper and show every step: the solutions are written out as methods, so that you can check your route and not just your final number. Name every result you lean on (“by the bias–variance decomposition”, “by the invariance of maximum likelihood”, “by the pivotal method”) and state the hypotheses it needs — normal population? known $σ$ ? independence? — each time you use it. No calculator is needed: every number has been chosen to come out exactly, and wherever a critical value is required it is given in the question. Full worked solutions are on a separate page; each question also has its own reveal, so you can check one answer without seeing the rest.

1.Computation
A population has mean $μ$ and variance $σ^{2} > 0$ , and $X_{1}, \dots, X_{n}$ is an i.i.d. sample from it with $n \geq 2$ . Three estimators of $μ$ are proposed:
$\overset{μ}{^}_{A} = \overset{ˉ}{X}, \overset{μ}{^}_{B} = \frac{X _{1} + X _{2}}{2}, \overset{μ}{^}_{C} = \frac{n}{n + 1} \overset{ˉ}{X} .$
1. (a)Find the bias and the variance of each estimator.
2. (b)Find the mean squared error of each, and show that $\overset{μ}{^}_{C}$ beats $\overset{ˉ}{X}$ on MSE exactly when $μ^{2} < (2 + \frac{1}{n}) σ^{2}$ . Then evaluate all three MSEs for $n = 4$ , $μ = 2$ , $σ = 2$ .
3. (c)Which of the three are consistent for $μ$ ? Where a theorem settles it, name the theorem; where an estimator is not consistent, argue directly from the definition.
Solution
(a) Expectations and variances. By the theorem Mean and variance of the sample mean, $E (\overset{ˉ}{X}) = μ$ and $Var (\overset{ˉ}{X}) = σ^{2} / n$ , so $\overset{μ}{^}_{A}$ has bias $0$ and variance $σ^{2} / n$ .
$\overset{μ}{^}_{B}$ is itself the sample mean of a sample of size $2$ , so $E (\overset{μ}{^}_{B}) = μ$ and the bias is $0$ ; by independence the variances add, $Var (\overset{μ}{^}_{B}) = \frac{1}{4} (σ^{2} + σ^{2}) = σ^{2} /2$ .
For $\overset{μ}{^}_{C}$ , linearity of expectation gives $E (\overset{μ}{^}_{C}) = \frac{n}{n + 1} μ$ , so
$Bias (\overset{μ}{^}_{C}) = \frac{n}{n + 1} μ - μ = - \frac{μ}{n + 1}, Var (\overset{μ}{^}_{C}) = (\frac{n}{n + 1})^{2} \frac{σ ^{2}}{n} = \frac{n σ ^{2}}{( n + 1 ) ^{2}},$
the constant coming out of the variance squared.
(b) Bias–variance decomposition. $MSE = Var + Bias^{2}$ , so
$MSE_{A} = \frac{σ ^{2}}{n}, MSE_{B} = \frac{σ ^{2}}{2}, MSE_{C} = \frac{n σ ^{2}}{( n + 1 ) ^{2}} + \frac{μ ^{2}}{( n + 1 ) ^{2}} = \frac{n σ ^{2} + μ ^{2}}{( n + 1 ) ^{2}} .$
$MSE_{C} < MSE_{A}$ if and only if $n (n σ^{2} + μ^{2}) < (n + 1)^{2} σ^{2}$ . Expanding the right side, $n^{2} σ^{2} + n μ^{2} < n^{2} σ^{2} + 2 n σ^{2} + σ^{2}$ , which is $n μ^{2} < (2 n + 1) σ^{2}$ , i.e. $μ^{2} < (2 + \frac{1}{n}) σ^{2}$ . Shrinking toward zero pays when the true mean is within about $2$ population standard deviations of zero: the bias it introduces is then smaller than the variance it removes.
With $n = 4$ , $μ = 2$ , $σ = 2$ : $MSE_{A} = 4/4 = 1$ , $MSE_{B} = 4/2 = 2$ , and $MSE_{C} = (16 + 4) /25 = 4/5$ . Check against the criterion: $μ^{2} = 4 < \frac{9}{4} \cdot 4 = 9$ ✓, so $\overset{μ}{^}_{C}$ should win, and it does.
(c) Consistency. $\overset{μ}{^}_{A}$ is consistent by the corollary The sample mean is consistent. For $\overset{μ}{^}_{C}$ , the bias $- μ / (n + 1) \to 0$ and the variance $n σ^{2} / (n + 1)^{2} \leq σ^{2} / n \to 0$ , so $MSE_{C} \to 0$ and the theorem MSE consistency gives consistency.
$\overset{μ}{^}_{B}$ has $MSE = σ^{2} /2$ , which does not vanish — but that alone proves nothing, because the theorem runs one way only (MSE to zero implies consistency). Argue from the definition. The estimator $\overset{μ}{^}_{B}$ does not depend on $n$ at all, so for each $ε > 0$ the number $c (ε) = P (∣ \overset{μ}{^}_{B} - μ ∣ \geq ε)$ is the same for every $n$ , and the limit as $n \to \infty$ is simply $c (ε)$ . If $c (ε) = 0$ for every $ε > 0$ , then taking $ε = 1/ k$ and a countable union gives $P (\overset{μ}{^}_{B} \neq = μ) = 0$ , so $Var (\overset{μ}{^}_{B}) = 0$ — contradicting $Var (\overset{μ}{^}_{B}) = σ^{2} /2 > 0$ . Hence some $ε$ has $c (ε) > 0$ , the probability does not tend to $0$ , and $\overset{μ}{^}_{B}$ is not consistent.
The moral: discarding $n - 2$ observations keeps unbiasedness perfectly intact and destroys consistency.
2.Proof
Two independent, unbiased estimators $\hat{θ}_{1}$ and $\hat{θ}_{2}$ of the same parameter $θ$ have variances $v_{1} > 0$ and $v_{2} > 0$ — think of the means of two independent surveys of different sizes. For a real number $w$ define the combination
$\hat{θ}_{w} = w \hat{θ}_{1} + (1 - w) \hat{θ}_{2} .$
1. (a)Prove that $\hat{θ}_{w}$ is unbiased for every $w$ , and find $Var (\hat{θ}_{w})$ as a function of $w$ .
2. (b)Prove that the variance is minimised at $w^{*} = \frac{v _{2}}{v _{1} + v _{2}}$ and that the minimum value is $\frac{v _{1} v _{2}}{v _{1} + v _{2}}$ .
3. (c)Prove that this minimum is strictly smaller than both $v_{1}$ and $v_{2}$ . Then take $\hat{θ}_{1} = \overset{ˉ}{X}_{1}$ and $\hat{θ}_{2} = \overset{ˉ}{X}_{2}$ , the means of independent samples of sizes $n_{1}$ and $n_{2}$ from one population with variance $σ^{2}$ : identify $\hat{θ}_{w^{*}}$ and its variance.
Solution
(a) Linearity. $E (\hat{θ}_{w}) = wE (\hat{θ}_{1}) + (1 - w) E (\hat{θ}_{2}) = wθ + (1 - w) θ = θ$ for every real $w$ — no restriction to $[0, 1]$ is needed for unbiasedness. Since the two estimators are independent, the variance of the sum is the sum of the variances, and constants come out squared:
$Var (\hat{θ}_{w}) = w^{2} v_{1} + (1 - w)^{2} v_{2} =: g (w) .$
(b) Minimise the parabola. $g^{'} (w) = 2 w v_{1} - 2 (1 - w) v_{2}$ , which vanishes when $w (v_{1} + v_{2}) = v_{2}$ , i.e. at $w^{*} = v_{2} / (v_{1} + v_{2})$ . Since $g^{''} (w) = 2 (v_{1} + v_{2}) > 0$ , $g$ is a convex parabola and the stationary point is its global minimum. With $1 - w^{*} = v_{1} / (v_{1} + v_{2})$ ,
$g (w^{*}) = \frac{v _{2}^{2} v _{1}}{( v _{1} + v _{2} ) ^{2}} + \frac{v _{1}^{2} v _{2}}{( v _{1} + v _{2} ) ^{2}} = \frac{v _{1} v _{2} ( v _{2} + v _{1} )}{( v _{1} + v _{2} ) ^{2}} = \frac{v _{1} v _{2}}{v _{1} + v _{2}} . ■$
Equivalently $g (w^{*}) = 1/ (1/ v_{1} + 1/ v_{2})$ : the precisions (reciprocal variances) add, and each estimator is weighted in proportion to its precision.
(c) The combination beats both. $\frac{v _{1} v _{2}}{v _{1} + v _{2}} < v_{1}$ is equivalent, after multiplying by the positive $v_{1} + v_{2}$ and dividing by $v_{1} > 0$ , to $v_{2} < v_{1} + v_{2}$ , i.e. $v_{1} > 0$ , which holds. By symmetry the minimum is also below $v_{2}$ . $■$
So throwing the noisier estimator away (which is $w = 1$ or $w = 0$ ) is never optimal: however poor it is, it carries some information, and the right weight extracts it. For instance if $v_{2} = 9 v_{1}$ the minimum is $9 v_{1}^{2} / (10 v_{1}) = 0.9 v_{1}$ , a $10%$ reduction bought from an estimator nine times noisier.
Sample means. Here $v_{i} = σ^{2} / n_{i}$ , so
$w^{*} = \frac{σ ^{2} / n _{2}}{σ ^{2} / n _{1} + σ ^{2} / n _{2}} = \frac{1/ n _{2}}{( n _{1} + n _{2} ) / ( n _{1} n _{2} )} = \frac{n _{1}}{n _{1} + n _{2}},$
and therefore
$\hat{θ}_{w^{*}} = \frac{n _{1} X ˉ _{1} + n _{2} X ˉ _{2}}{n _{1} + n _{2}} = \frac{sum of all n _{1} + n _{2} observations}{n _{1} + n _{2}},$
the grand mean of the pooled sample. Its variance is $σ^{2} / (n_{1} + n_{2})$ , which agrees with the formula: $\frac{( σ ^{2} / n _{1} ) ( σ ^{2} / n _{2} )}{σ ^{2} / n _{1} + σ ^{2} / n _{2}} = \frac{σ ^{2}}{n _{1} + n _{2}}$ ✓. Sample means are to be weighted by sample size, not averaged equally.
3.Computation
Lifetimes $X_{1}, \dots, X_{n}$ are modelled as i.i.d. exponential with rate $λ > 0$ , density $f (x; λ) = λ e^{- λ x}$ for $x \geq 0$ , for which $E (X) = 1/ λ$ . Three observed lifetimes are $2$ , $3$ and $7$ .
1. (a)Find the method of moments estimator of $λ$ and evaluate it on the data.
2. (b)Find the maximum likelihood estimator of $λ$ : write the log-likelihood, solve the score equation, and confirm that the stationary point is the global maximum over $λ > 0$ .
3. (c)By naming the theorem that allows it, write down the MLE of the mean lifetime $1/ λ$ and of the survival probability $P (X > 6) = e^{- 6 λ}$ . Then decide, with justification, whether the MLE of the mean is unbiased and whether the MLE of $λ$ is unbiased.
Solution
(a) Match the first moment. One parameter needs one equation: set the population mean equal to the sample mean, $1/ λ = \overset{x}{ˉ}$ , so $\tilde{λ} = 1/ \overset{x}{ˉ}$ . On the data $\overset{x}{ˉ} = (2 + 3 + 7) /3 = 4$ and $\tilde{λ} = 1/4$ .
(b) Maximum likelihood. The likelihood is
$L (λ) = i = 1 \prod n λ e^{- λ x_{i}} = λ^{n} e^{- λ \sum x_{i}}, ℓ (λ) = n lo g λ - λ i = 1 \sum n x_{i} .$
The score is $ℓ^{'} (λ) = n / λ - \sum x_{i}$ , which vanishes at $\hat{λ} = n / \sum x_{i} = 1/ \overset{x}{ˉ}$ . Second derivative: $ℓ^{''} (λ) = - n / λ^{2} < 0$ for every $λ > 0$ , so $ℓ$ is strictly concave on $(0, \infty)$ ; a strictly concave function has at most one stationary point and it is the global maximum. The boundary check confirms it: $ℓ \to - \infty$ as $λ \to 0^{+}$ (the $n lo g λ$ term) and as $λ \to \infty$ (the $- λ \sum x_{i}$ term dominates), so nothing larger hides at the edges. On the data $\hat{λ} = 3/12 = 1/4$ , agreeing with the method of moments.
(c) Invariance. By the theorem Invariance of maximum likelihood, the MLE of any function of $λ$ is that function of $\hat{λ}$ . So the MLE of the mean $1/ λ$ is $1/ \hat{λ} = \overset{x}{ˉ} = 4$ , and the MLE of $P (X > 6) = e^{- 6 λ}$ is $e^{- 6/4} = e^{- 3/2}$ .
Is $\overset{ˉ}{X}$ unbiased for $1/ λ$ ? Yes: $E (\overset{ˉ}{X}) = E (X) = 1/ λ$ by linearity.
Is $\hat{λ} = 1/ \overset{ˉ}{X}$ unbiased for $λ$ ? No. The map $u \mapsto 1/ u$ is strictly convex on $(0, \infty)$ and $\overset{ˉ}{X}$ is not a constant, so Jensen's inequality gives $E (1/ \overset{ˉ}{X}) > 1/ E (\overset{ˉ}{X}) = λ$ : the MLE of the rate is biased upward. (One can show $E (\hat{λ}) = \frac{n}{n - 1} λ$ for $n \geq 2$ , so for these three observations the estimator overshoots by $50%$ on average.) Invariance transports the maximiser through $g$ ; it says nothing about expectations, which is why unbiasedness has no invariance property.
4.Sketch
Four observations from a Uniform $(0, θ)$ population are $1, 2, 3$ and $10$ . The density is $f (x; θ) = 1/ θ$ for $0 \leq x \leq θ$ and $0$ otherwise.
1. (a)Write the likelihood $L (θ)$ as an explicit piecewise formula valid for every $θ > 0$ , and explain why it is zero on part of its domain.
2. (b)Sketch $L (θ)$ for $0 < θ \leq 20$ , measuring the vertical axis in units of $1 0^{- 4}$ . Mark the maximum likelihood estimate and the value of $L$ there, plot the exact values at $θ = 12$ , $15$ and $20$ , and mark where the method of moments estimate $2 \overset{x}{ˉ}$ falls. Every feature you draw must be justified.
3. (c)Explain why solving $ℓ^{'} (θ) = 0$ produces nothing here, and say which step of the maximum likelihood recipe locates the estimate instead.
4. (d)Given that $E (X_{(n)}) = \frac{n}{n + 1} θ$ for the sample maximum, produce an unbiased estimator from the MLE and evaluate it. Explain why it must sit to the right of the MLE on your sketch.
Solution
(a) The likelihood is a product of indicator-like factors. Each factor $f (x_{i}; θ)$ equals $1/ θ$ if $θ \geq x_{i}$ and $0$ otherwise, so the product is nonzero only when $θ$ is at least every observation, i.e. $θ \geq max_{i} x_{i} = 10$ :
$L (θ) = {0, θ^{- 4}, 0 < θ < 10, θ \geq 10.$
It is zero below $10$ because a Uniform $(0, θ)$ population with $θ < 10$ cannot produce the observation $10$ : such a $θ$ assigns the observed data density zero, and a single zero factor kills the whole product.
(b) The sketch. Axes: $θ$ from $0$ to $20$ horizontally, $L$ vertically in units of $1 0^{- 4}$ , so that $L (10) = 1 0^{- 4}$ is $1$ unit.
Zero segment. From $θ = 0$ to $θ = 10$ the graph runs along the horizontal axis.
Jump at $θ = 10$ . At $θ = 10$ the value is $1 0^{- 4} = 1$ unit, and it belongs to the curve (a filled dot), because $x \leq θ$ is allowed with equality. So the graph jumps vertically from $0$ to $1$ unit; the point just to the left is at height $0$ .
Decreasing tail. For $θ > 10$ , $L (θ) = θ^{- 4}$ is strictly decreasing ( $L^{'} = - 4 θ^{- 5} < 0$ ) and convex ( $L^{''} = 20 θ^{- 6} > 0$ ), bending toward the axis without reaching it. Exact values, in units of $1 0^{- 4}$ : $L (12) = (10/12)^{4} = (5/6)^{4} = 625/1296 \approx 0.48$ , $L (15) = (2/3)^{4} = 16/81 \approx 0.20$ , $L (20) = (1/2)^{4} = 1/16 = 0.0625$ .
The MLE. The highest point of the graph is the top of the jump, at $θ = 10$ : mark $\hat{θ} = 10$ with $L (\hat{θ}) = 1 0^{- 4}$ .
The method of moments estimate. $\overset{x}{ˉ} = (1 + 2 + 3 + 10) /4 = 4$ , so $2 \overset{x}{ˉ} = 8$ . Mark $θ = 8$ on the axis; there $L (8) = 0$ . The sketch shows at a glance what is wrong with it: the estimate lies in the region the data have ruled out, since the observed $10$ cannot come from Uniform $(0, 8)$ .
(c) No stationary point. On $θ \geq 10$ , $ℓ (θ) = - 4 lo g θ$ and $ℓ^{'} (θ) = - 4/ θ < 0$ , never zero; below $10$ the likelihood is identically zero and $ℓ$ is undefined. So "differentiate and solve" returns nothing. It is step 4 of the recipe, check the boundary, that finds the answer: the parameter values compatible with the data form $[10, \infty)$ , $L$ is decreasing there, so the maximum is at the left endpoint $\hat{θ} = x_{(4)} = 10$ , a point where the derivative is not zero because the graph has a jump, not a peak.
(d) The unbiased correction. From $E (X_{(n)}) = \frac{n}{n + 1} θ$ , the estimator $\frac{n + 1}{n} X_{(n)}$ has expectation $θ$ . With $n = 4$ it is $\frac{5}{4} \cdot 10 = 12.5$ .
It lies to the right of the MLE because the correction multiplies by $\frac{n + 1}{n} > 1$ , and the reason it should is visible on the sketch: the maximum of four draws can never exceed $θ$ , so it systematically falls short of the true ceiling and must be inflated to be right on average. On the graph, $12.5$ sits on the decreasing tail at height $(10/12.5)^{4} = (4/5)^{4} = 256/625 \approx 0.41$ units — a lower likelihood than the MLE, but a positive one. The two estimators disagree because they optimise different things: fit to the data in hand, against correctness on average over repeated samples.
5.Computation
Let $X_{1}, \dots, X_{n}$ be i.i.d. Uniform $(0, θ)$ with $n \geq 2$ . Three estimators of $θ$ are in play: the maximum likelihood estimator $\hat{θ}_{ML} = X_{(n)} = max_{i} X_{i}$ , its unbiased correction $\hat{θ}_{U} = \frac{n + 1}{n} X_{(n)}$ , and the method of moments estimator $\hat{θ}_{MM} = 2 \overset{ˉ}{X}$ . You may use that $X_{(n)}$ has density $g (x) = \frac{n x ^{n - 1}}{θ ^{n}}$ on $[0, θ]$ , and that a Uniform $(0, θ)$ variable has variance $θ^{2} /12$ .
1. (a)Compute $E (X_{(n)})$ and $E (X_{(n)}^{2})$ , and hence show that $Var (X_{(n)}) = \frac{n θ ^{2}}{( n + 1 ) ^{2} ( n + 2 )}$ .
2. (b)Find the mean squared error of each of the three estimators as an exact function of $n$ and $θ$ .
3. (c)Rank the three by MSE for $n \geq 3$ , proving each inequality, and find the one value of $n$ at which two of them tie. Evaluate all three MSEs for $n = 4$ , $θ = 10$ .
4. (d)The Cramér–Rao bound says an unbiased estimator's variance is at least $1/ (n I (θ))$ , which is of order $1/ n$ — yet $\hat{θ}_{U}$ is unbiased with variance of order $1/ n^{2}$ . Identify the hypothesis of the theorem that fails for this model, and demonstrate the failure by computing the expectation of the score.
Solution
(a) Moments of the maximum. Integrate against the given density:
$E (X_{(n)}) = \int_{0}^{θ} x \cdot \frac{n x ^{n - 1}}{θ ^{n}} d x = \frac{n}{θ ^{n}} \cdot \frac{θ ^{n + 1}}{n + 1} = \frac{n θ}{n + 1}, E (X_{(n)}^{2}) = \frac{n}{θ ^{n}} \cdot \frac{θ ^{n + 2}}{n + 2} = \frac{n θ ^{2}}{n + 2} .$
Then
$Var (X_{(n)}) = \frac{n θ ^{2}}{n + 2} - \frac{n ^{2} θ ^{2}}{( n + 1 ) ^{2}} = n θ^{2} \frac{( n + 1 ) ^{2} - n ( n + 2 )}{( n + 2 ) ( n + 1 ) ^{2}} = \frac{n θ ^{2}}{( n + 1 ) ^{2} ( n + 2 )},$
since $(n + 1)^{2} - n (n + 2) = n^{2} + 2 n + 1 - n^{2} - 2 n = 1$ .
(b) Three MSEs by the bias–variance decomposition.
MLE. Bias $= \frac{n θ}{n + 1} - θ = - \frac{θ}{n + 1}$ , so
$MSE_{ML} = \frac{n θ ^{2}}{( n + 1 ) ^{2} ( n + 2 )} + \frac{θ ^{2}}{( n + 1 ) ^{2}} = \frac{θ ^{2} ( n + n + 2 )}{( n + 1 ) ^{2} ( n + 2 )} = \frac{2 θ ^{2}}{( n + 1 ) ( n + 2 )} .$
Unbiased correction. Zero bias, and the constant comes out squared:
$MSE_{U} = (\frac{n + 1}{n})^{2} \frac{n θ ^{2}}{( n + 1 ) ^{2} ( n + 2 )} = \frac{θ ^{2}}{n ( n + 2 )} .$
Method of moments. $E (2 \overset{ˉ}{X}) = 2 \cdot θ /2 = θ$ , unbiased, and $Var (2 \overset{ˉ}{X}) = 4 Var (\overset{ˉ}{X}) = 4 \cdot \frac{θ ^{2} /12}{n}$ , so $MSE_{MM} = \frac{θ ^{2}}{3 n}$ .
(c) Ranking. $MSE_{U} < MSE_{ML}$ iff $\frac{1}{n ( n + 2 )} < \frac{2}{( n + 1 ) ( n + 2 )}$ iff $n + 1 < 2 n$ iff $n > 1$ : true for every $n \geq 2$ . $MSE_{ML} < MSE_{MM}$ iff $\frac{2}{( n + 1 ) ( n + 2 )} < \frac{1}{3 n}$ iff $6 n < n^{2} + 3 n + 2$ iff $n^{2} - 3 n + 2 > 0$ iff $(n - 1) (n - 2) > 0$ iff $n \geq 3$ . So for $n \geq 3$ ,
$MSE_{U} < MSE_{ML} < MSE_{MM},$
and the tie is at $n = 2$ , where $MSE_{ML} = 2 θ^{2} /12 = θ^{2} /6 = MSE_{MM}$ .
With $n = 4$ , $θ = 10$ : $MSE_{U} = 100/24 = 25/6$ , $MSE_{ML} = 200/30 = 20/3$ , $MSE_{MM} = 100/12 = 25/3$ . The corrected maximum has half the error of the method of moments, and the gap widens with $n$ : the maximum-based errors fall like $1/ n^{2}$ , the moment-based error like $1/ n$ , so the ratio grows like $n /3$ .
(d) The regularity conditions fail. The Cramér–Rao theorem rests on the lemma The score has mean zero, which is proved by differentiating $\int f (x; θ) d x = 1$ under the integral sign. That step needs the support of $f$ not to depend on $θ$ ; here the support is $[0, θ]$ , and moving $θ$ moves the endpoint of the integral, producing a boundary term the lemma ignores. The failure is checkable: on the support, $lo g f (x; θ) = - lo g θ$ , so the score is $\partial_{θ} lo g f = - 1/ θ$ , a constant, and
$E [\partial_{θ} lo g f (X; θ)] = - \frac{1}{θ} \neq = 0.$
The first conclusion of the lemma is false for this model, so the bound built on it does not apply, and there is nothing to stop an unbiased estimator from converging at the faster rate $1/ n^{2}$ . This is the general pattern: when the parameter sets the edge of the support, the extreme observations carry far more information than any average of the sample.
6.Computation
In each model below a single observation $X$ has the stated mass function. Work with one observation first, then with an i.i.d. sample of size $n$ .
1. (a)Poisson: $f (x; λ) = e^{- λ} λ^{x} / x!$ for $x = 0, 1, 2, \dots$ , with $E (X) = Var (X) = λ$ . Find the score, verify directly that it has mean zero, and compute $I (λ)$ twice — once as the variance of the score and once as the expected curvature — checking that the two agree.
2. (b)State the Cramér–Rao bound for unbiased estimators of $λ$ from $n$ observations, and show that $\overset{ˉ}{X}$ attains it.
3. (c)Bernoulli: $f (x; p) = p^{x} (1 - p)^{1 - x}$ for $x \in {0, 1}$ . Show that $I (p) = \frac{1}{p ( 1 - p )}$ and that $\overset{p}{^} = \overset{ˉ}{X}$ is efficient.
4. (d)In the Poisson model, $T = \frac{1}{2} (X_{1} + \overset{ˉ}{X})$ is unbiased for $λ$ . Find its efficiency as a function of $n$ , check that the value at $n = 1$ is what it must be, and describe what happens as $n \to \infty$ .
Solution
(a) Poisson. $lo g f (x; λ) = - λ + x lo g λ - lo g x!$ , so the score is
$\frac{\partial}{\partial λ} lo g f = \frac{x}{λ} - 1.$
Its mean is $E (X) / λ - 1 = λ / λ - 1 = 0$ , as the lemma The score has mean zero promises. As a variance: $I (λ) = Var (X / λ - 1) = Var (X) / λ^{2} = λ / λ^{2} = 1/ λ$ . As curvature: $\partial_{λ}^{2} lo g f = - x / λ^{2}$ , so $- E [\partial_{λ}^{2} lo g f] = E (X) / λ^{2} = 1/ λ$ . The two agree ✓. Note the information decreases as $λ$ grows: a Poisson with large mean is also a Poisson with large spread, and each count pins $λ$ down less tightly.
(b) The bound and $\overset{ˉ}{X}$ . By the theorem Cramér–Rao lower bound, every unbiased estimator of $λ$ has $Var (\hat{λ}) \geq \frac{1}{n I ( λ )} = \frac{λ}{n}$ . The sample mean is unbiased, $E (\overset{ˉ}{X}) = λ$ , with $Var (\overset{ˉ}{X}) = Var (X) / n = λ / n$ by the theorem Mean and variance of the sample mean. It sits exactly on the bound, so it is efficient: no unbiased estimator of a Poisson mean can do better.
(c) Bernoulli. $lo g f (x; p) = x lo g p + (1 - x) lo g (1 - p)$ , so
$\partial_{p} lo g f = \frac{x}{p} - \frac{1 - x}{1 - p}, \partial_{p}^{2} lo g f = - \frac{x}{p ^{2}} - \frac{1 - x}{( 1 - p ) ^{2}} .$
Using $E (X) = p$ and $E (1 - X) = 1 - p$ ,
$I (p) = - E [\partial_{p}^{2} lo g f] = \frac{p}{p ^{2}} + \frac{1 - p}{( 1 - p ) ^{2}} = \frac{1}{p} + \frac{1}{1 - p} = \frac{1}{p ( 1 - p )} .$
The bound is $\frac{1}{n I ( p )} = \frac{p ( 1 - p )}{n}$ , and $\overset{p}{^} = \overset{ˉ}{X}$ is unbiased with variance $Var (X) / n = p (1 - p) / n$ . Efficient. (Information is smallest at $p = \frac{1}{2}$ , where a coin is least predictable, and blows up near $0$ and $1$ .)
(d) A deliberately lopsided estimator. Unbiasedness: $E (T) = \frac{1}{2} (λ + λ) = λ$ . For the variance, $X_{1}$ and $\overset{ˉ}{X}$ are not independent, since $X_{1}$ is one of the terms in $\overset{ˉ}{X}$ :
$Cov (X_{1}, \overset{ˉ}{X}) = \frac{1}{n} j = 1 \sum n Cov (X_{1}, X_{j}) = \frac{1}{n} Var (X_{1}) = \frac{λ}{n},$
the other covariances vanishing by independence. Hence
$Var (T) = \frac{1}{4} [Var (X_{1}) + Var (\overset{ˉ}{X}) + 2 Cov (X_{1}, \overset{ˉ}{X})] = \frac{1}{4} [λ + \frac{λ}{n} + \frac{2 λ}{n}] = \frac{λ ( n + 3 )}{4 n} .$
The efficiency is the bound over the actual variance:
$eff (T) = \frac{λ / n}{λ ( n + 3 ) / ( 4 n )} = \frac{4}{n + 3} .$
At $n = 1$ this is $1$ , as it must be: then $\overset{ˉ}{X} = X_{1}$ , $T = X_{1}$ , and $T$ is the efficient estimator itself. At $n = 5$ the efficiency is $\frac{1}{2}$ , and as $n \to \infty$ it tends to $0$ — $T$ keeps half its weight on a single observation forever, so its variance never falls below $λ /4$ while the bound falls like $1/ n$ . An estimator can be unbiased for every $n$ and still waste almost all of a large sample.
7.Proof
Let $X_{1}, \dots, X_{n}$ be i.i.d. $N (μ, σ^{2})$ and write $v = σ^{2}$ for the parameter of interest. You may quote three facts: for $Z \sim N (0, 1)$ , $E (Z^{4}) = 3$ ; a $χ_{k}^{2}$ variable has mean $k$ and variance $2 k$ ; and $(n - 1) S^{2} / σ^{2} \sim χ_{n - 1}^{2}$ .
1. (a)With $μ$ known, prove that $I (v) = \frac{1}{2 v ^{2}}$ , and state the Cramér–Rao bound for unbiased estimators of $σ^{2}$ .
2. (b)Prove that $\overset{v}{^}_{0} = \frac{1}{n} \sum_{i = 1}^{n} (X_{i} - μ)^{2}$ is unbiased and attains the bound.
3. (c)Now let $μ$ be unknown. Compute $Var (S^{2})$ and the efficiency of $S^{2}$ relative to the bound in (a). What has estimating $μ$ cost?
4. (d)Prove that the maximum likelihood estimator $\overset{σ}{^}^{2} = \frac{1}{n} \sum_{i = 1}^{n} (X_{i} - \overset{ˉ}{X})^{2} = \frac{n - 1}{n} S^{2}$ has strictly smaller mean squared error than $S^{2}$ for every $n \geq 2$ , despite being biased.
Solution
(a) Information about a variance. For one observation, $lo g f (x; v) = - \frac{1}{2} lo g (2 π) - \frac{1}{2} lo g v - \frac{( x - μ ) ^{2}}{2 v}$ . Differentiating in $v$ ,
$\partial_{v} lo g f = - \frac{1}{2 v} + \frac{( x - μ ) ^{2}}{2 v ^{2}}, \partial_{v}^{2} lo g f = \frac{1}{2 v ^{2}} - \frac{( x - μ ) ^{2}}{v ^{3}} .$
Since $E [(X - μ)^{2}] = v$ , the score has mean $- \frac{1}{2 v} + \frac{v}{2 v ^{2}} = 0$ (as the lemma requires), and by the curvature form of the lemma The score has mean zero, and information as curvature,
$I (v) = - E [\partial_{v}^{2} lo g f] = - \frac{1}{2 v ^{2}} + \frac{v}{v ^{3}} = \frac{1}{2 v ^{2}} . ■$
The theorem Cramér–Rao lower bound then says every unbiased estimator of $σ^{2}$ satisfies $Var \geq \frac{1}{n I ( v )} = \frac{2 v ^{2}}{n} = \frac{2 σ ^{4}}{n}$ .
(b) The known-mean estimator is efficient. Unbiasedness is linearity: $E (\overset{v}{^}_{0}) = \frac{1}{n} \sum E [(X_{i} - μ)^{2}] = v$ . For the variance, write $(X_{i} - μ)^{2} = v Z_{i}^{2}$ with $Z_{i} = (X_{i} - μ) / σ \sim N (0, 1)$ . Then $Var (Z_{i}^{2}) = E (Z_{i}^{4}) - (E Z_{i}^{2})^{2} = 3 - 1 = 2$ , so $Var ((X_{i} - μ)^{2}) = 2 v^{2}$ , and by independence
$Var (\overset{v}{^}_{0}) = \frac{1}{n ^{2}} \cdot n \cdot 2 v^{2} = \frac{2 v ^{2}}{n},$
exactly the bound. $\overset{v}{^}_{0}$ is efficient. $■$
(c) The price of not knowing $μ$ . Write $S^{2} = \frac{σ ^{2}}{n - 1} V$ with $V \sim χ_{n - 1}^{2}$ . Then $E (S^{2}) = σ^{2}$ (unbiased, as known) and
$Var (S^{2}) = \frac{σ ^{4}}{( n - 1 ) ^{2}} \cdot 2 (n - 1) = \frac{2 σ ^{4}}{n - 1} .$
Its efficiency relative to the bound of (a) is
$\frac{2 σ ^{4} / n}{2 σ ^{4} / ( n - 1 )} = \frac{n - 1}{n} < 1.$
$S^{2}$ behaves as if it had one observation fewer: the variance it achieves with $n$ data points is what $\overset{v}{^}_{0}$ achieves with $n - 1$ . Estimating $μ$ has cost exactly one degree of freedom — the same one that appears in the divisor $n - 1$ and in the $t_{n - 1}$ distribution. The loss vanishes as $n \to \infty$ .
(d) The biased MLE wins on MSE. $\overset{σ}{^}^{2} = \frac{n - 1}{n} S^{2}$ has $E (\overset{σ}{^}^{2}) = \frac{n - 1}{n} σ^{2}$ , so its bias is $- σ^{2} / n$ , and
$Var (\overset{σ}{^}^{2}) = (\frac{n - 1}{n})^{2} \frac{2 σ ^{4}}{n - 1} = \frac{2 ( n - 1 ) σ ^{4}}{n ^{2}} .$
By the theorem Bias–variance decomposition,
$MSE (\overset{σ}{^}^{2}) = \frac{2 ( n - 1 ) σ ^{4}}{n ^{2}} + \frac{σ ^{4}}{n ^{2}} = \frac{( 2 n - 1 ) σ ^{4}}{n ^{2}}, MSE (S^{2}) = Var (S^{2}) = \frac{2 σ ^{4}}{n - 1} .$
Now $\frac{2 n - 1}{n ^{2}} < \frac{2}{n - 1}$ iff $(2 n - 1) (n - 1) < 2 n^{2}$ (both denominators positive) iff $2 n^{2} - 3 n + 1 < 2 n^{2}$ iff $1 < 3 n$ , which holds for every $n \geq 1$ . So the inequality is strict for all $n \geq 2$ . $■$
At $n = 2$ the biased estimator's MSE is $\frac{3}{4} σ^{4}$ against $2 σ^{4}$ ; at $n = 10$ it is $0.19 σ^{4}$ against $\frac{2}{9} σ^{4} \approx 0.22 σ^{4}$ . Neither divisor is actually MSE-optimal: minimising $MSE (c \sum (X_{i} - \overset{ˉ}{X})^{2}) = σ^{4} [2 c^{2} (n - 1) + (c (n - 1) - 1)^{2}]$ over $c$ gives $c = 1/ (n + 1)$ , so the best estimator in this family divides by $n + 1$ and is more biased still. Unbiasedness is a constraint, and constraints cost.
8.Critique
A machine's fill volumes are normal with known $σ = 8$ ml. A sample of $n = 16$ bottles has $\overset{x}{ˉ} = 50$ ml. Use $z_{0.025} = 1.96$ and $z_{0.05} = 1.645$ . A student submits the following.
Step 1. The standard error is $σ / n = 8/4 = 2$ .
Step 2. The $95%$ interval is $50 \pm 1.96 \times 2 = (46.08, 53.92)$ .
Step 3. Therefore the probability that $μ$ lies in $(46.08, 53.92)$ is $0.95$ .
Step 4. Since the interval captures $95%$ of the distribution, about $95%$ of individual bottles contain between $46.08$ and $53.92$ ml.
Step 5. The $90%$ interval is $50 \pm 1.645 \times 2 = (46.71, 53.29)$ . It is narrower, so it is a more precise estimate of $μ$ and should be preferred.
Step 6. If a second sample of $16$ bottles were taken, its mean would fall inside $(46.08, 53.92)$ with probability $0.95$ .
1. (a)Steps 1 and 2 are correct. For each of Steps 3–6, say whether it is correct; where it is not, name precisely what is wrong and write the statement that is correct.
2. (b)Suppose $μ$ were in fact $50$ . What interval contains the middle $95%$ of individual fill volumes, and by what factor is it wider than the confidence interval? Explain why the confidence interval shrinks to a point as $n$ grows while this one does not.
3. (c)For Step 6, write the correct probability as a statement about a standard normal variable, and show without tables that it is less than $0.95$ .
Solution
(a) Four claims, four verdicts.
Step 3 is wrong: it assigns a probability to a constant. In the pivotal argument $P (\overset{ˉ}{X} - 1.96 σ / n \leq μ \leq \overset{ˉ}{X} + 1.96 σ / n) = 0.95$ , the random quantities are the endpoints; $μ$ is a fixed number. Once the sample is in, $(46.08, 53.92)$ is a fixed interval too, and it either contains $μ$ or it does not — there is no randomness left and so no probability to assign. Correct statement: "the interval was produced by a procedure which, over repeated samples, captures $μ$ $95%$ of the time." The $95%$ describes the procedure, not this interval.
Step 4 is wrong: it confuses the precision of an estimate of $μ$ with the spread of the population. The interval's half-width is $1.96 σ / n$ , which measures how well $\overset{ˉ}{X}$ locates $μ$ ; individual bottles scatter with standard deviation $σ$ , four times larger here. Correct statement: the interval says nothing about individual bottles; see (b).
Step 5 is wrong: narrowing by lowering the level is not precision, it is a lower success rate. At fixed $n$ and $σ$ the only thing a $90%$ interval buys with its shorter length is that the procedure misses $μ$ twice as often — $10%$ of the time rather than $5%$ . Correct statement: the $90%$ interval is $(46.71, 53.29)$ , and it is not a better estimate but a riskier one. Genuine precision at a fixed level comes only from a larger $n$ or a smaller $σ$ .
Step 6 is wrong: the interval is about $μ$ , not about future data. A second sample mean $\overset{ˉ}{X}^{'}$ is itself random with standard deviation $σ / n = 2$ , and the interval's centre $\overset{ˉ}{X}$ is random as well; the gap $\overset{ˉ}{X}^{'} - \overset{ˉ}{X}$ has two sources of variability. The correct probability is computed in (c) and is well below $0.95$ .
(b) Population band versus confidence interval. If $μ = 50$ , an individual volume $X \sim N (50, 64)$ lies in $μ \pm 1.96 σ = 50 \pm 15.68 = (34.32, 65.68)$ with probability $0.95$ . Its width is $31.36$ ml against the confidence interval's $7.84$ ml: a factor of $n = 4$ , because the interval's half-width is $1.96 σ / n$ while the band's is $1.96 σ$ .
As $n$ grows, $σ / n \to 0$ and the confidence interval collapses onto $μ$ — more data locates the mean ever more precisely. The population band is $μ \pm 1.96 σ$ whatever $n$ is: no amount of sampling makes individual bottles less variable. The two intervals answer different questions, and only the first is what a confidence interval is for.
(c) The chance a second mean lands in the interval. Let $\overset{ˉ}{X}^{'}$ be the second sample mean, independent of $\overset{ˉ}{X}$ , with $\overset{ˉ}{X}, \overset{ˉ}{X}^{'} \sim N (μ, 4)$ . Then $\overset{ˉ}{X}^{'} - \overset{ˉ}{X} \sim N (0, 4 + 4) = N (0, 8)$ , with standard deviation $22$ . The event in Step 6 is $∣ \overset{ˉ}{X}^{'} - \overset{ˉ}{X} ∣ \leq 3.92$ , so
$P (∣ \overset{ˉ}{X}^{'} - \overset{ˉ}{X} ∣ \leq 3.92) = P (∣ Z ∣ \leq \frac{3.92}{2 2}) = P (∣ Z ∣ \leq \frac{1.96}{2}) .$
Since $1.96/ 2 < 1.96$ and $c \mapsto P (∣ Z ∣ \leq c)$ is increasing, this is strictly less than $P (∣ Z ∣ \leq 1.96) = 0.95$ . (From a table it is about $0.83$ .) To capture the next sample mean with probability $0.95$ one would need the wider margin $1.96 \cdot 22$ , a prediction interval — a different object from a confidence interval.
9.Computation
A normal population has unknown mean $μ$ and unknown variance. A sample of $n = 5$ gives the values $7, 9, 10, 11, 13$ . Use $t_{0.025, 4} = 2.776$ , $t_{0.005, 4} = 4.604$ and $z_{0.025} = 1.96$ .
1. (a)Compute $\overset{x}{ˉ}$ , $s^{2}$ and the estimated standard error $s / n$ exactly.
2. (b)State the pivot, name its distribution and the two facts about normal samples that the distribution rests on, and construct the $95%$ confidence interval for $μ$ .
3. (c)Construct the $99%$ interval, and explain why it must be wider.
4. (d)A classmate uses $z$ in place of $t$ and reports $10 \pm 1.96 = (8.04, 11.96)$ . Write this interval's true coverage as a probability about a $t_{4}$ variable, and explain in terms of the pivot why it is below $95%$ .
Solution
(a) Summary statistics. The sum is $50$ , so $\overset{x}{ˉ} = 10$ . The deviations are $- 3, - 1, 0, 1, 3$ , with squares $9, 1, 0, 1, 9$ summing to $20$ , so
$s^{2} = \frac{20}{n - 1} = \frac{20}{4} = 5, s = 5, \frac{s}{n} = \frac{5}{5} = 1.$
(b) The $t$ pivot. Because $σ$ is unknown, the pivot is the studentised mean
$T = \frac{X ˉ - μ}{S / n} \sim t_{n - 1} = t_{4},$
by the theorem The studentised mean is pivotal. That distribution rests on two facts about a normal sample: $(n - 1) S^{2} / σ^{2} \sim χ_{n - 1}^{2}$ , and $\overset{ˉ}{X}$ and $S^{2}$ are independent — which is what lets $T$ be written as $Z / V / (n - 1)$ with $Z$ and $V$ independent, the definition of Student's $t$ . Both hypotheses need the population to be normal, which is given.
By the pivotal method, $P (- 2.776 \leq T \leq 2.776) = 0.95$ ; multiplying through by $S / n$ and rearranging around $μ$ gives $μ \in \overset{ˉ}{X} \pm 2.776 S / n$ . With $\overset{x}{ˉ} = 10$ and $s / n = 1$ ,
$(10 - 2.776, 10 + 2.776) = (7.224, 12.776) .$
(c) The $99%$ interval replaces $t_{0.025, 4}$ by $t_{0.005, 4}$ :
$10 \pm 4.604 = (5.396, 14.604) .$
It must be wider because $t_{0.005, 4} > t_{0.025, 4}$ : to trap the pivot with probability $0.99$ instead of $0.95$ the central region must extend further into the tails, and with only $4$ degrees of freedom those tails are heavy, so the cost is large — the interval is $1.66$ times as long. Confidence is bought with length at a fixed $n$ .
(d) What the $z$ -interval really covers. The interval $\overset{ˉ}{X} \pm 1.96 S / n$ contains $μ$ exactly when $∣ T ∣ \leq 1.96$ , where $T$ is the same pivot as in (b). Its true coverage is therefore
$P (∣ T ∣ \leq 1.96), T \sim t_{4},$
and since $1.96 < 2.776$ and $c \mapsto P (∣ T ∣ \leq c)$ is increasing, this is strictly less than $P (∣ T ∣ \leq 2.776) = 0.95$ . (It is about $0.88$ .) The classmate's error is to treat $s$ as if it were the known $σ$ : with $n = 5$ the estimate $S$ varies a great deal from sample to sample, the pivot has the heavy tails of $t_{4}$ rather than the normal's, and a margin sized for the normal traps $μ$ too rarely. The interval is not "more precise" — it is under-covering.
10.Proof
Independent samples of sizes $n_{1}$ and $n_{2}$ are drawn from $N (μ_{1}, σ_{1}^{2})$ and $N (μ_{2}, σ_{2}^{2})$ , where both variances are known and need not be equal. Write $δ = μ_{1} - μ_{2}$ and $z = z_{α /2}$ .
1. (a)Derive, by the pivotal method, a $100 (1 - α) %$ confidence interval for $δ$ : state the distribution of $\overset{ˉ}{X}_{1} - \overset{ˉ}{X}_{2}$ with justification, exhibit the pivot, and carry out the rearrangement.
2. (b)Suppose the total budget $N = n_{1} + n_{2}$ is fixed. Prove that the interval's half-width is minimised when $n_{1} : n_{2} = σ_{1} : σ_{2}$ , and show that the minimal half-width is $z (σ_{1} + σ_{2}) / N$ .
3. (c)Prove that equal allocation $n_{1} = n_{2} = N /2$ inflates $Var (\overset{ˉ}{X}_{1} - \overset{ˉ}{X}_{2})$ by the factor $\frac{2 ( σ _{1}^{2} + σ _{2}^{2} )}{( σ _{1} + σ _{2} ) ^{2}} \geq 1$ , with equality if and only if $σ_{1} = σ_{2}$ . Evaluate the factor when $σ_{1} = 3 σ_{2}$ and translate it into extra observations.
Solution
(a) The pivot. By the proposition A normal population gives an exactly normal mean, $\overset{ˉ}{X}_{1} \sim N (μ_{1}, σ_{1}^{2} / n_{1})$ and $\overset{ˉ}{X}_{2} \sim N (μ_{2}, σ_{2}^{2} / n_{2})$ , and they are independent because the samples are. A difference of independent normals is normal, and by the lemma Linearity of expectation and variance of an independent sum,
$\overset{ˉ}{X}_{1} - \overset{ˉ}{X}_{2} \sim N (δ, τ^{2}), τ^{2} = \frac{σ _{1}^{2}}{n _{1}} + \frac{σ _{2}^{2}}{n _{2}},$
where $τ$ is a known constant. Standardising,
$Q = \frac{( X ˉ _{1} - X ˉ _{2} ) - δ}{τ} \sim N (0, 1),$
a distribution involving no unknown at all — a pivot. By definition of $z$ , $P (- z \leq Q \leq z) = 1 - α$ . Multiply the chain by $τ > 0$ , subtract $\overset{ˉ}{X}_{1} - \overset{ˉ}{X}_{2}$ , and multiply by $- 1$ , which reverses both inequalities and swaps the ends:
$(\overset{ˉ}{X}_{1} - \overset{ˉ}{X}_{2}) - z τ \leq δ \leq (\overset{ˉ}{X}_{1} - \overset{ˉ}{X}_{2}) + z τ .$
The event is unchanged, so its probability is still $1 - α$ : the interval is $(\overset{ˉ}{X}_{1} - \overset{ˉ}{X}_{2}) \pm z σ_{1}^{2} / n_{1} + σ_{2}^{2} / n_{2}$ . $■$
(b) Optimal allocation. The half-width is $z g (n_{1})$ with
$g (n_{1}) = \frac{σ _{1}^{2}}{n _{1}} + \frac{σ _{2}^{2}}{N - n _{1}}, 0 < n_{1} < N,$
treating $n_{1}$ as continuous; since the square root is increasing, minimise $g$ . Then
$g^{'} (n_{1}) = - \frac{σ _{1}^{2}}{n _{1}^{2}} + \frac{σ _{2}^{2}}{( N - n _{1} ) ^{2}},$
which vanishes iff $σ_{2}^{2} n_{1}^{2} = σ_{1}^{2} (N - n_{1})^{2}$ , i.e. (all quantities positive) $σ_{2} n_{1} = σ_{1} (N - n_{1})$ , i.e. $n_{1} / n_{2} = σ_{1} / σ_{2}$ . This is the global minimum because $g^{''} (n_{1}) = 2 σ_{1}^{2} / n_{1}^{3} + 2 σ_{2}^{2} / (N - n_{1})^{3} > 0$ makes $g$ convex on $(0, N)$ , and $g \to \infty$ at both ends of the interval. Explicitly $n_{1}^{*} = \frac{N σ _{1}}{σ _{1} + σ _{2}}$ , $n_{2}^{*} = \frac{N σ _{2}}{σ _{1} + σ _{2}}$ , and
$g (n_{1}^{*}) = \frac{σ _{1}^{2} ( σ _{1} + σ _{2} )}{N σ _{1}} + \frac{σ _{2}^{2} ( σ _{1} + σ _{2} )}{N σ _{2}} = \frac{( σ _{1} + σ _{2} ) ^{2}}{N},$
so the minimal half-width is $z (σ_{1} + σ_{2}) / N$ . $■$ (An alternative route is the Cauchy–Schwarz inequality, $(\frac{σ _{1}^{2}}{n _{1}} + \frac{σ _{2}^{2}}{n _{2}}) (n_{1} + n_{2}) \geq (σ_{1} + σ_{2})^{2}$ , with equality exactly when $σ_{i} \propto n_{i}$ . In practice $n_{1}^{*}$ is rounded to an integer.) The rule says: spend the budget in proportion to the standard deviations, not the variances — the noisier population gets more observations, but only linearly more.
(c) The cost of splitting evenly. With $n_{1} = n_{2} = N /2$ , $g = \frac{2 σ _{1}^{2}}{N} + \frac{2 σ _{2}^{2}}{N} = \frac{2 ( σ _{1}^{2} + σ _{2}^{2} )}{N}$ , and dividing by the optimum $(σ_{1} + σ_{2})^{2} / N$ gives the stated factor. It is at least $1$ because
$2 (σ_{1}^{2} + σ_{2}^{2}) - (σ_{1} + σ_{2})^{2} = σ_{1}^{2} - 2 σ_{1} σ_{2} + σ_{2}^{2} = (σ_{1} - σ_{2})^{2} \geq 0,$
with equality iff $σ_{1} = σ_{2}$ . $■$
With $σ_{1} = 3 σ_{2}$ the factor is $\frac{2 ( 9 + 1 ) σ _{2}^{2}}{16 σ _{2}^{2}} = \frac{20}{16} = \frac{5}{4}$ . Since the variance scales as $1/ N$ , an even split needs $\frac{5}{4} N$ observations — $25%$ more — to match the width the optimal split achieves with $N$ , which puts three quarters of the budget on the noisier population. When the variances are equal the optimum is the even split, which is why the pooled $t$ interval, whose hypotheses assume a common variance, is at its best with $n_{1} = n_{2}$ .
11.Computation
Throughout this question take $z_{α /2} = 2$ — the value $1.96$ rounded, so that every number comes out exactly; the level is then approximately $95%$ . A sample of $n = 16$ independent trials is observed.
1. (a)With $k = 4$ successes, compute the Wald interval and the Wilson score interval for $p$ exactly.
2. (b)The Wilson interval is by definition the set of $p$ satisfying $∣ \overset{p}{^} - p ∣ \leq z p (1 - p) / n$ . Verify by direct substitution that both Wilson endpoints from (a) satisfy this with equality. Then show that the lower Wald endpoint fails the inequality while the upper one satisfies it strictly, and say what that means about the Wald interval.
3. (c)With $k = 0$ successes, compute both intervals, verify the upper Wilson endpoint with equality, and say which interval is an honest report of what $16$ trials established.
4. (d)Prove that with $z = 2$ the Wilson centre is exactly $\frac{k + 2}{n + 4}$ for every $k$ and $n$ , and interpret the formula.
Solution
(a) Two intervals at $\overset{p}{^} = 1/4$ . Here $\overset{p}{^} = 4/16 = \frac{1}{4}$ and $\overset{p}{^} (1 - \overset{p}{^}) / n = \frac{1}{4} \cdot \frac{3}{4} \cdot \frac{1}{16} = \frac{3}{256}$ .
Wald. $3/256 = 3 /16$ , so the interval is $\frac{1}{4} \pm 2 \cdot 3 /16 = \frac{1}{4} \pm 3 /8$ . Since $3 \approx 1.73$ , this is about $(0.033, 0.467)$ .
Wilson. With $z^{2} = 4$ : $z^{2} / n = \frac{1}{4}$ , $z^{2} / (2 n) = \frac{1}{8}$ , $z^{2} / (4 n^{2}) = 4/1024 = \frac{1}{256}$ . The centre is
$\tilde{p} = \frac{\frac{1}{4} + \frac{1}{8}}{1 + \frac{1}{4}} = \frac{3/8}{5/4} = \frac{3}{10},$
and the half-width is
$w = \frac{2}{5/4} \frac{3}{256} + \frac{1}{256} = \frac{8}{5} \frac{4}{256} = \frac{8}{5} \cdot \frac{2}{16} = \frac{8}{5} \cdot \frac{1}{8} = \frac{1}{5} .$
The Wilson interval is $(\frac{3}{10} - \frac{1}{5}, \frac{3}{10} + \frac{1}{5}) = (\frac{1}{10}, \frac{1}{2})$ . It is shifted toward $\frac{1}{2}$ relative to Wald and slightly longer.
(b) Checking the defining inequality. At $p = \frac{1}{2}$ : left side $∣ \frac{1}{4} - \frac{1}{2} ∣ = \frac{1}{4}$ ; right side $2 \frac{1}{2} \cdot \frac{1}{2} /16 = 2 1/64 = \frac{2}{8} = \frac{1}{4}$ . Equality ✓. At $p = \frac{1}{10}$ : left side $∣ \frac{1}{4} - \frac{1}{10} ∣ = \frac{5}{20} - \frac{2}{20} = \frac{3}{20}$ ; right side $2 \frac{1}{10} \cdot \frac{9}{10} /16 = 2 9/1600 = 2 \cdot \frac{3}{40} = \frac{3}{20}$ . Equality ✓. (At $p = \overset{p}{^} = \frac{1}{4}$ the left side is $0$ , so $\overset{p}{^}$ is inside, as it must be.)
The Wald endpoints solve a different equation, $∣ \overset{p}{^} - p ∣ = z \overset{p}{^} (1 - \overset{p}{^}) / n$ , whose right side is a constant $3 /8$ rather than a function of $p$ . Consider the lower one, $p_{L} = \frac{1}{4} - \frac{3}{8}$ . Since $0 < p_{L} < \frac{1}{4} < \frac{1}{2}$ and $p (1 - p)$ is increasing on $[0, \frac{1}{2}]$ , we have $p_{L} (1 - p_{L}) < \frac{1}{4} \cdot \frac{3}{4}$ , so the honest right side $2 p_{L} (1 - p_{L}) /16$ is smaller than $2 3/256 = 3 /8 = ∣ \overset{p}{^} - p_{L} ∣$ : the inequality fails at $p_{L}$ , which lies outside the honest set. For the upper one, $p_{U} = \frac{1}{4} + \frac{3}{8}$ , we have $\frac{1}{4} < p_{U} < \frac{1}{2}$ (as $3 /8 < \frac{1}{4}$ ), so $p_{U} (1 - p_{U}) > \frac{3}{16}$ and the honest right side exceeds $∣ \overset{p}{^} - p_{U} ∣$ : the inequality holds strictly, and $p_{U}$ is an interior point of the honest set. So Wald reaches too far down and stops too short going up — it is the honest interval $(\frac{1}{10}, \frac{1}{2})$ shifted the wrong way, because it pretends the standard error is the same at every $p$ .
(c) No successes. $\overset{p}{^} = 0$ . Wald: $\overset{p}{^} (1 - \overset{p}{^}) / n = 0$ , so the interval is $0 \pm 0 = {0}$ . Wilson: $\tilde{p} = \frac{0 + \frac{1}{8}}{5/4} = \frac{1}{10}$ and $w = \frac{8}{5} 0 + \frac{1}{256} = \frac{8}{5} \cdot \frac{1}{16} = \frac{1}{10}$ , giving $(0, \frac{1}{5})$ .
Check $p = \frac{1}{5}$ : left side $∣0 - \frac{1}{5} ∣ = \frac{1}{5}$ ; right side $2 \frac{1}{5} \cdot \frac{4}{5} /16 = 2 4/400 = 2 \cdot \frac{2}{20} = \frac{1}{5}$ . Equality ✓. (At $p = 0$ both sides are $0$ , so the lower endpoint is exactly $0$ , as it should be when nothing was observed.)
The honest report is Wilson's. Zero events in $16$ trials is entirely consistent with a rate of $\frac{1}{5}$ : the chance of seeing none is then $(4/5)^{16} \approx 0.03$ , exactly the sort of borderline value an endpoint of a $95%$ interval should have. Wald's ${0}$ asserts, with $95%$ confidence, that the rate is exactly zero on the strength of $16$ trials — a zero-width interval from a finite sample, which is not a confidence statement at all. (The rule of three gives $3/16 \approx 0.19$ as the upper bound, close to Wilson's $0.2$ .)
(d) Add two successes and two failures. With $\overset{p}{^} = k / n$ and $z^{2} = 4$ ,
$\tilde{p} = \frac{\frac{k}{n} + \frac{4}{2 n}}{1 + \frac{4}{n}} = \frac{( k + 2 ) / n}{( n + 4 ) / n} = \frac{k + 2}{n + 4} . ■$
Check against (a) and (c): $(4 + 2) / (16 + 4) = 6/20 = \frac{3}{10}$ ✓ and $2/20 = \frac{1}{10}$ ✓. The Wilson centre is the sample proportion after two phantom successes and two phantom failures are appended: a shrinkage of $\overset{p}{^}$ toward $\frac{1}{2}$ whose weight $4$ is fixed while the data's weight $n$ grows, so it matters at $n = 16$ and vanishes as $n \to \infty$ , where Wilson collapses back onto Wald. It is precisely this pull away from the endpoints that keeps the interval inside $[0, 1]$ and gives it positive width at $\overset{p}{^} = 0$ .
12.Critique
A polling firm's planning memo reads as follows. Take $z_{0.025} = 2$ and $z_{0.005} = 2.6$ (both rounded) throughout, and measure margins as proportions, so that “ $5$ points” means $E = 0.05$ .
Baseline. Our last poll used $n = 400$ . At the worst case $p = \frac{1}{2}$ the $95%$ margin is $2 \frac{1}{4} \cdot \frac{1}{400} = \frac{1}{20}$ , i.e. $5$ points, which is what we reported.
Claim A. To halve the margin to $2.5$ points, double the sample: $n = 800$ .
Claim B. The next poll is in a town of $50, 000$ rather than a country of $5$ million. The population is $100$ times smaller, so $n = 4$ gives the same $5$ -point margin.
Claim C. The client wants $99%$ confidence at the same $5$ -point margin. As $99/95 \approx 1.04$ , about $4%$ more respondents will do: $n \approx 416$ .
Claim D. Our pilot puts support near $10%$ . Using $p = 0.1$ , $n = \frac{2 ^{2} \times 0.1 \times 0.9}{0.0 5 ^{2}} = 144$ respondents guarantee a $5$ -point margin.
1. (a)The baseline is correct. For each of Claims A–D, decide whether it is right; where it is wrong, name the error precisely and compute the correct sample size.
2. (b)Prove that if it is known that $p \leq p_{0}$ for some $p_{0} < \frac{1}{2}$ , then $n \geq z^{2} p_{0} (1 - p_{0}) / E^{2}$ guarantees a margin of at most $E$ , and that this requirement never exceeds the worst-case one. Then find the true $p$ at which Claim D's $n = 144$ delivers a margin of exactly $\frac{1}{15}$ (about $6.7$ points), and comment.
3. (c)The firm settles on $n = 1600$ at $95%$ and reports a $2.5$ -point margin. Its respondents were recruited from the firm's own website. Does the margin account for this? Say what a confidence interval does and does not measure.
Solution
(a) Four claims.
Claim A is wrong: the margin scales as $1/ n$ , not $1/ n$ . The margin is $z p (1 - p) / n$ , so halving it needs $n$ multiplied by $4$ : the correct figure is $n = 1600$ . Check: $2 \frac{1}{4} \cdot \frac{1}{1600} = 2 \cdot \frac{1}{2} \cdot \frac{1}{40} = \frac{1}{40}$ , i.e. $2.5$ points ✓. Doubling to $800$ would give $2 \frac{1}{4} \cdot \frac{1}{800} = \frac{1}{800} = \frac{1}{20 2}$ , about $3.5$ points — the quadratic cost of precision, by the proposition Sample size for a target margin of error.
Claim B is wrong: the population size does not appear anywhere. The formula $n \geq z^{2} / (4 E^{2})$ contains $z$ and $E$ and nothing else, because the model is $n$ independent draws from a population large enough that each draw is a fresh Bernoulli $(p)$ trial; that holds for a town of $50, 000$ as well as a country. The correct figure is $n = 400$ again. Taken at its word, $n = 4$ gives a margin of $2 \frac{1}{4} \cdot \frac{1}{4} = \frac{1}{2}$ — fifty points, which is no poll at all.
Claim C is wrong: $n$ scales with the square of the critical value, not with the confidence level. By the proposition Worst-case sample size for a proportion,
$n \geq \frac{z _{0.005}^{2}}{4 E ^{2}} = \frac{6.76}{4 \times 0.0025} = \frac{6.76}{0.01} = 676.$
The factor over the $95%$ figure is $(2.6/2)^{2} = 1.69$ , so $69%$ more respondents, not $4%$ . (With the unrounded $2.576/1.96$ it is about $1.73$ .)
Claim D is half right. The arithmetic is correct: $4 \times 0.09/0.0025 = 0.36/0.0025 = 144$ . The word "guarantee" is not. With $n = 144$ the margin is $2 p (1 - p) /144 = p (1 - p) /6$ , which is at most $0.05$ only if $p (1 - p) \leq 0.09$ , i.e. only if $p \leq 0.1$ or $p \geq 0.9$ . A pilot figure is an estimate, not a bound; if support is really $15%$ or $20%$ the promise is broken. Correct planning takes the worst case over the range of $p$ that is genuinely plausible — for instance if $p \leq 0.2$ can be relied on, $n \geq 4 \times 0.16/0.0025 = 256$ — or falls back to $400$ if nothing can.
(b) Planning with a bound on $p$ . Let $g (p) = p (1 - p)$ . Then $g^{'} (p) = 1 - 2 p > 0$ on $[0, \frac{1}{2})$ , so $g$ is increasing there, and $p \leq p_{0} < \frac{1}{2}$ implies $p (1 - p) \leq p_{0} (1 - p_{0})$ . Hence the margin obeys
$z \frac{p ( 1 - p )}{n} \leq z \frac{p _{0} ( 1 - p _{0} )}{n} \leq E as soon as n \geq \frac{z ^{2} p _{0} ( 1 - p _{0} )}{E ^{2}} .$
Since $p_{0} (1 - p_{0}) \leq \frac{1}{4}$ (the maximum of $g$ on $[0, 1]$ is at $\frac{1}{2}$ ), this is never more than the worst-case $z^{2} / (4 E^{2})$ . $■$ Knowledge that $p$ is far from $\frac{1}{2}$ is worth real money, but only knowledge of a bound, not a point guess.
For Claim D's $n = 144$ , the margin equals $\frac{1}{15}$ when $p (1 - p) /6 = \frac{1}{15}$ , i.e. $p (1 - p) = \frac{6}{15} = \frac{2}{5}$ , i.e. $p (1 - p) = \frac{4}{25}$ . Clearing denominators, $25 p^{2} - 25 p + 4 = 0$ , which factors as $(5 p - 1) (5 p - 4) = 0$ , so $p = \frac{1}{5}$ or $p = \frac{4}{5}$ . If true support is $20%$ rather than the pilot's $10%$ , the advertised $5$ -point margin is really $6.7$ points — and a small pilot being ten points off is not unusual.
(c) What the margin measures. No. The margin of error quantifies sampling variability only: how much $\overset{p}{^}$ would move across repeated random samples of $1600$ from the population the model describes. Visitors to the firm's website are self-selected, so the sample is drawn from a different population than the electorate, and the interval is a precise statement about that wrong population. Raising $n$ from $400$ to $1600$ narrows the interval without moving it toward the truth; the bias is invisible to the formula and is fixed only by the sampling design. The interval does measure the uncertainty from having $1600$ observations rather than all of them; it does not measure, and cannot measure, non-response, self-selection, or a leading question.

← Back to the question sheet