Sampling and Data Distributions

(a) Parameter, statistic, or neither. (i) $\mu =E(X)$ is a fixed numerical feature of the population: a parameter. It does not change from sample to sample and is not known from the data.

(ii) $\bar{X}=\frac{1}{2}(X_1+X_2)$ can be computed from the data alone: a statistic. It is also a random variable — it takes a different value on a different sample.

(iii) Neither. It is not a parameter, because it depends on which sample was drawn; and it is not a statistic, because evaluating it needs $\mu$, which the data do not supply. (Replacing $\mu$ by $\bar{X}$ turns it into a statistic.)

(iv) The sample maximum is a function of the observations alone: a statistic.

(v) $\sigma /\sqrt{n}$ is a fixed constant determined by the population (and the chosen $n$); it is not random and cannot be computed from a sample. It is a parameter — specifically the standard error of $\bar{X}$ — and the statistic that estimates it is $S/\sqrt{n}$.

(b) The population and the sampling distribution. The population mean is

$$\mu =\frac{2+4+6+8}{4}=5,$$

and the population variance, dividing by $N=4$ because this is the whole population and not a sample, is

$${\sigma }^2=\frac{(-3{)}^2+(-1{)}^2+1^2+3^2}{4}=\frac{9+1+1+9}{4}=\frac{20}{4}=5.$$

The $16$ ordered samples $(a,b)$ are equally likely, each with probability $\frac{1}{16}$, and $\bar{x}=\frac{a+b}{2}$. Arranging them in a grid with $a$ down the rows and $b$ across the columns, the sample means are

$$\begin{array}{ccccc}& 2& 4& 6& 8\\ 2& 2& 3& 4& 5\\ 4& 3& 4& 5& 6\\ 6& 4& 5& 6& 7\\ 8& 5& 6& 7& 8\end{array}$$

Counting along the anti-diagonals, the sampling distribution of $\bar{X}$ is

$$\begin{array}{cccccccc}\bar{x}& 2& 3& 4& 5& 6& 7& 8\\ P(\bar{X}=\bar{x})& \frac{1}{16}& \frac{2}{16}& \frac{3}{16}& \frac{4}{16}& \frac{3}{16}& \frac{2}{16}& \frac{1}{16}\end{array}$$

— a triangle centred on $5$. The probabilities sum to $\frac{16}{16}=1$.

(c) Mean and variance from the table.

$$E(\bar{X})=\frac{2\cdot 1+3\cdot 2+4\cdot 3+5\cdot 4+6\cdot 3+7\cdot 2+8\cdot 1}{16}=\frac{2+6+12+20+18+14+8}{16}=\frac{80}{16}=5=\mu .$$

For the variance, the deviations $\bar{x}-5$ are $-3,-2,-1,0,1,2,3$ with frequencies $1,2,3,4,3,2,1$, so

$$\mathrm{Var}(\bar{X})=\frac{1\cdot 9+2\cdot 4+3\cdot 1+4\cdot 0+3\cdot 1+2\cdot 4+1\cdot 9}{16}=\frac{40}{16}=\frac{5}{2}=\frac{{\sigma }^2}{2}=\frac{{\sigma }^2}{n}.$$

Both theorems check. Note that the averaging halved the variance ($5$ to $\frac{5}{2}$) — this is the whole reason for taking a mean — and that the independence of the two draws is what licensed the ${\sigma }^2/n$; the equality $E(\bar{X})=\mu$ would have held even for dependent draws.

(a) The two formulas. With $\bar{x}=\frac{a+b}{2}$ the deviations are

$$a-\bar{x}=\frac{a-b}{2},b-\bar{x}=\frac{b-a}{2},$$

equal and opposite, as deviations from the sample mean must be (they always sum to zero). Each squares to $\frac{(a-b{)}^2}{4}$, so the sum of squared deviations is $\frac{(a-b{)}^2}{2}$. Dividing by $n-1=1$ gives $S^2=\frac{(a-b{)}^2}{2}$; dividing by $n=2$ gives ${\hat{\sigma }}^2=\frac{(a-b{)}^2}{4}$.

(b) The expectations. Over the $16$ ordered samples, $|a-b|$ takes the value $0$ on the $4$ diagonal samples $(2,2),(4,4),(6,6),(8,8)$; the value $2$ on the $6$ samples $(2,4),(4,2),(4,6),(6,4),(6,8),(8,6)$; the value $4$ on the $4$ samples $(2,6),(6,2),(4,8),(8,4)$; and the value $6$ on the $2$ samples $(2,8),(8,2)$. Check: $4+6+4+2=16$. Hence

$$\sum _{\text{samples}}(a-b{)}^2=4\cdot 0+6\cdot 4+4\cdot 16+2\cdot 36=0+24+64+72=160,E\left[(a-b{)}^2\right]=\frac{160}{16}=10.$$

(Cross-check without the table: $a-b$ has mean $0$ and, being a difference of independent draws, variance ${\sigma }^2+{\sigma }^2=10$, so $E[(a-b{)}^2]=\mathrm{Var}(a-b)=10$. The table agrees.)

Therefore

$$E(S^2)=\frac{10}{2}=5={\sigma }^2,E({\hat{\sigma }}^2)=\frac{10}{4}=\frac{5}{2}.$$

$S^2$ is unbiased. The divide-by-$n$ estimator is biased downward by the factor $\frac{n-1}{n}=\frac{1}{2}$: on average it reports half the true variance. The reason is visible in the diagonal samples, where both observations coincide with $\bar{x}$ and the estimator reports $0$; deviations from the sample's own centre are systematically too small.

(c) $E(S)$. Since $S=\sqrt{S^2}=\frac{|a-b|}{\sqrt{2}}$,

$$E|a-b|=\frac{4\cdot 0+6\cdot 2+4\cdot 4+2\cdot 6}{16}=\frac{12+16+12}{16}=\frac{40}{16}=\frac{5}{2},E(S)=\frac{5}{2\sqrt{2}}=\frac{5\sqrt{2}}{4}.$$

Compare with $\sigma =\sqrt{5}$ by squaring both (both are positive): ${\left(\frac{5\sqrt{2}}{4}\right)}^2=\frac{50}{16}=\frac{25}{8}=3.125<5$. So $E(S)<\sigma$ — roughly $1.77$ against $2.24$.

The general fact: $S^2$ is unbiased for ${\sigma }^2$, but $S$ is not unbiased for $\sigma$. Unbiasedness is not preserved by taking square roots, because the square root is strictly concave and $E(\sqrt{S^2})<\sqrt{E(S^2)}$ whenever $S^2$ is not constant. The sample standard deviation underestimates $\sigma$ on average.

(a) Decompose the variance. The stratum weights are $W_1=W_2=\frac{1000}{2000}=\frac{1}{2}$, so

$$\mu =\frac{1}{2}(12)+\frac{1}{2}(28)=20.$$

The within-strata component is ${\sum }_h{W}_h{\sigma }_h^2=\frac{1}{2}(36)+\frac{1}{2}(36)=36$. The between-strata component is

$$\sum _h{W}_h({\mu }_h-\mu {)}^2=\frac{1}{2}(12-20{)}^2+\frac{1}{2}(28-20{)}^2=\frac{1}{2}(64)+\frac{1}{2}(64)=64.$$

By the law of total variance applied to the stratum label, ${\sigma }^2=36+64=100$, so $\sigma =10$. Almost two-thirds of the population variance is between the groups.

(b) The two standard errors. Under simple random sampling (treating $n/N=0.05$ as small enough to ignore the finite population correction),

$$\mathrm{Var}({\bar{X}}_{\text{SRS}})=\frac{{\sigma }^2}{n}=\frac{100}{100}=1,SE=1\text{ km}.$$

Proportional allocation takes $n_h=n{W}_h=100\cdot \frac{1}{2}=50$ from each stratum. The strata are sampled independently, so

$$\mathrm{Var}({\bar{X}}_{\text{str}})=\sum _h{W}_h^2\frac{{\sigma }_h^2}{n_h}=\frac{1}{4}\cdot \frac{36}{50}+\frac{1}{4}\cdot \frac{36}{50}=\frac{36}{100}=0.36,$$

which is exactly $\frac{1}{n}{\sum }_h{W}_h{\sigma }_h^2=\frac{36}{100}$, the within-strata variance divided by $n$. Hence $SE=\sqrt{0.36}=0.6$ km. Stratifying cut the standard error from $1$ to $0.6$ for the same hundred interviews.

(c) The price in interviews, and the source of the gain. An SRS matches $0.36$ when $\frac{100}{n}\le 0.36$, i.e. $n\ge \frac{100}{0.36}=\frac{2500}{9}\approx 277.8$, so $278$ interviews — nearly three times as many.

The difference between the two variances is exactly the between-strata term divided by $n$: $1-0.36=0.64=\frac{64}{100}$. If both strata had mean $20$, the between-strata component would be $0$, ${\sigma }^2$ would be $36$, and both designs would give $\mathrm{Var}(\bar{X})=\frac{36}{100}$. Stratification would then buy nothing. Stratification deletes the between-strata variance and only that, so the variable you stratify on must be one that separates the stratum means; strata that differ only in spread are worthless.

(a) Where the argument fails. Two design failures, neither of them arithmetical.

First, undercoverage: the sampling frame is "people who visit the council website", not "residents of the city". Residents who never visit the site cannot be selected at all.

Second, voluntary response bias: nobody selected these $50,000$ — they selected themselves, and people who feel strongly (here, drivers who would pay the toll) are far more likely to bother. The sample is also a convenience sample from the council's point of view.

Both failures shift $E(\hat{p})$ away from $p$: the survey estimates, quite precisely, the proportion of self-selected website visitors who oppose the toll, which is a different parameter from the one the council wants.

Step 2's number is the standard error $\hat{p}$ would have if the $50,000$ were an i.i.d. Bernoulli$(p)$ sample. The formula's hypothesis fails, so the number describes nothing.

The false claim in Step 3 is that the interval $(0.776,0.784)$ contains the population proportion. The phrase doing the damage is "no sampling error". That phrase is nearly true — sampling variability really is tiny at $n=50,000$ — but variability is not the error here. The error is bias, which the margin of error does not measure at all. The report has a very narrow interval around the wrong centre: a precise answer to the wrong question.

(b) Why more responses cannot help. Sampling variability shrinks like $1/\sqrt{n}$: quadrupling $n$ halves the standard error, from about $0.0019$ to about $0.0009$. Bias is $E(\hat{p})-p$, and it does not depend on $n$ at all: the extra $150,000$ self-selected responses come from the same distorted frame with the same self-selection, so they estimate the same wrong quantity. The councillor would obtain a still narrower interval around the same wrong centre. (This is the Literary Digest mistake: $2.4$ million responses, off by nearly twenty points.)

(c) A legitimate design. Draw a simple random sample from a complete frame of residents — the electoral roll or the register of residential addresses — with the council choosing the respondents, each with equal probability, rather than the respondents choosing themselves. (Stratifying by district would do even better.) Then the responses are i.i.d. Bernoulli$(p)$ and the formulas of Steps 2 and 3 apply.

With $n=400$ and $p^{\ast }=0.5$,

$$SE=\sqrt{\frac{0.5\times 0.5}{400}}=\sqrt{\frac{1}{1600}}=\frac{1}{40}=0.025,ME=1.96\times 0.025=0.049,$$

about $\pm 5$ points. Four hundred properly chosen residents give an honest $\pm 5\%$; fifty thousand self-selected ones give a meaningless $\pm 0.4\%$.

The remaining hazard is nonresponse bias: some of the $400$ selected residents will decline, and if decliners differ from responders the estimate shifts again. Plan for it in advance — a follow-up protocol on a random subsample of the non-responders — rather than hoping it away.

(a) Mean and variance of $\bar{X}$. The mean. By linearity of expectation, pulling the constant $\frac{1}{n}$ out of the sum,

$$E(\bar{X})=\frac{1}{n}\sum _{i=1}^{n}E(X_i)=\frac{1}{n}\cdot n\mu =\mu .$$

Only the fact that every $X_i$ has mean $\mu$ was used; independence plays no role.

The variance. Write $D_i=X_i-\mu$, so $E(D_i)=0$, $E(D_i^2)={\sigma }^2$ and $E(D_i{D}_j)=\mathrm{Cov}(X_i,X_j)$. Then $\sum X_i-n\mu =\sum D_i$, and expanding the square of the sum (which is the two-variable expansion applied term by term),

$$\mathrm{Var}\left(\sum _{i=1}^n{X}_i\right)=E\left[(\sum _i{D}_i{)}^2\right]=\sum _{i}E(D_i^2)+\sum _{i\ne j}E(D_i{D}_j)=n{\sigma }^2+\sum _{i\ne j}\mathrm{Cov}(X_i,X_j).$$

Independence is used here, and only here: independent variables have $E(X_i{X}_j)=E(X_i)E(X_j)$, so every covariance is $0$ and the double sum vanishes, leaving $\mathrm{Var}(\sum X_i)=n{\sigma }^2$. Finally, scaling by the constant $\frac{1}{n}$ multiplies a variance by $\frac{1}{n^2}$:

$$\mathrm{Var}(\bar{X})=\frac{1}{n^2}\cdot n{\sigma }^2=\frac{{\sigma }^2}{n}.■$$

Without independence the mean result survives intact, and the variance becomes $\frac{{\sigma }^2}{n}+\frac{1}{n^2}{\sum }_{i\ne j}\mathrm{Cov}(X_i,X_j)$ — the formula ${\sigma }^2/n$ is exactly the uncorrelated case.

(b) Equicorrelated observations. Return to the general expansion of (a). There are $n(n-1)$ ordered pairs $(i,j)$ with $i\ne j$, and each contributes $\rho {\sigma }^2$, so

$$\mathrm{Var}\left(\sum _{i=1}^n{X}_i\right)=n{\sigma }^2+n(n-1)\rho {\sigma }^2=n{\sigma }^2(1+(n-1)\rho ).$$

Dividing by $n^2$,

$$\mathrm{Var}(\bar{X})=\frac{{\sigma }^2}{n}(1+(n-1)\rho ).■$$

Sanity checks: $\rho =0$ recovers ${\sigma }^2/n$; $\rho =1$ (all observations identical) gives ${\sigma }^2$, the variance of a single observation, which is right because $n$ copies of one number carry one number's worth of information.

(c) Consequences. A variance is never negative and ${\sigma }^2/n>0$, so (b) forces $1+(n-1)\rho \ge 0$, that is $\rho \ge -\frac{1}{n-1}$. Observations cannot be strongly negatively correlated with all of each other at once; for large $n$ the common correlation is essentially non-negative.

For $\rho >0$, split the formula:

$$\mathrm{Var}(\bar{X})=\frac{{\sigma }^2}{n}+\rho {\sigma }^2\cdot \frac{n-1}{n}⟶0+\rho {\sigma }^2=\rho {\sigma }^2>0(n\to \infty ).$$

The standard error has a floor of $\sigma \sqrt{\rho }$ that no amount of further sampling from the same cluster can breach: once the shared component of the cluster has been learned, extra units only repeat it.

Someone quoting $\sigma /\sqrt{n}$ for such data understates the true standard error by the factor $\sqrt{1+(n-1)\rho }$. With $\rho =0.1$ and $n=100$ that factor is $\sqrt{10.9}>3$: the reported uncertainty is less than a third of the real one, and the confidence intervals built on it are far too narrow. The correct hypothesis for ${\sigma }^2/n$ is uncorrelated observations, and a cluster design breaks exactly that.

(a) The square-root law. For i.i.d. observations $SE(\bar{X})=\sigma /\sqrt{n}$, so at $n=36$

$$SE=\frac{12}{\sqrt{36}}=\frac{12}{6}=2.$$

For $SE\le \frac{1}{2}$ we need $\frac{12}{\sqrt{n}}\le \frac{1}{2}$, i.e. $\sqrt{n}\ge 24$, i.e. $n\ge 576$. The smallest such $n$ is $576$. The standard error fell by a factor of $4$ (from $2$ to $\frac{1}{2}$) and the sample grew by a factor of $\frac{576}{36}=16=4^2$: dividing the standard error by $k$ costs a factor $k^2$ in sample size, because $n$ sits under a square root.

(b) Finite population correction. Without replacement from $N=81$,

$$\mathrm{Var}(\bar{X})=\frac{{\sigma }^2}{n}\cdot \frac{N-n}{N-1}=\frac{144}{36}\cdot \frac{81-36}{81-1}=4\cdot \frac{45}{80}=4\cdot \frac{9}{16}=\frac{9}{4},SE=\frac{3}{2}.$$

The convention is to ignore the correction when $n/N\le 0.05$. Here $n/N=\frac{36}{81}=\frac{4}{9}\approx 0.44$, so it is not applicable: ignoring it would report $2$ instead of $1.5$, overstating the uncertainty by a third. Sampling without replacement is more precise than the i.i.d. formula says, because later draws cannot repeat information already collected.

At $n=N$ the factor $\frac{N-n}{N-1}$ is $0$, so $SE=0$. That is correct: a sample of the whole population is a census, $\bar{X}$ equals $\mu$ identically, and there is no sampling variability left to measure.

(c) Enumeration check. Drawing $2$ of the $4$ values without replacement, the $6$ unordered samples are equally likely (each ordered sample has probability $\frac{1}{4}\cdot \frac{1}{3}=\frac{1}{12}$, and each unordered sample corresponds to two of them). Their means:

$$\{2,4\}\to 3,\{2,6\}\to 4,\{2,8\}\to 5,\{4,6\}\to 5,\{4,8\}\to 6,\{6,8\}\to 7.$$

The mean of the six values is $\frac{3+4+5+5+6+7}{6}=\frac{30}{6}=5=\mu$, so $\bar{X}$ is still unbiased. The deviations from $5$ are $-2,-1,0,0,1,2$, with squares $4,1,0,0,1,4$ summing to $10$, so

$$\mathrm{Var}(\bar{X})=\frac{10}{6}=\frac{5}{3}.$$

The formula predicts

$$\frac{{\sigma }^2}{n}\cdot \frac{N-n}{N-1}=\frac{5}{2}\cdot \frac{4-2}{4-1}=\frac{5}{2}\cdot \frac{2}{3}=\frac{5}{3},$$

in exact agreement. With replacement the same population gives $\mathrm{Var}(\bar{X})=\frac{5}{2}$; forbidding repeats removed the two extreme samples $(2,2)$ and $(8,8)$ and their kin, and the variance dropped by the factor $\frac{2}{3}$.

(a) The identity. Insert and remove $c$ inside the square: $X_i-\bar{X}=(X_i-c)-(\bar{X}-c)$. Squaring and summing over $i$,

$$\sum _{i=1}^n(X_i-\bar{X}{)}^2=\sum _{i=1}^n(X_i-c{)}^2-2(\bar{X}-c)\sum _{i=1}^n(X_i-c)+n(\bar{X}-c{)}^2,$$

where $(\bar{X}-c)$ came out of the middle sum because it does not depend on $i$. Now ${\sum }_i(X_i-c)={\sum }_i{X}_i-nc=n\bar{X}-nc=n(\bar{X}-c)$, so the middle term is $-2n(\bar{X}-c{)}^2$, and

$$\sum _{i=1}^n(X_i-\bar{X}{)}^2=\sum _{i=1}^n(X_i-c{)}^2-2n(\bar{X}-c{)}^2+n(\bar{X}-c{)}^2=\sum _{i=1}^n(X_i-c{)}^2-n(\bar{X}-c{)}^2.■$$

Rearranged, ${\sum }_i(X_i-c{)}^2={\sum }_i(X_i-\bar{X}{)}^2+n(\bar{X}-c{)}^2\ge {\sum }_i(X_i-\bar{X}{)}^2$, with equality exactly when $c=\bar{X}$. So the sample mean is the constant about which the sum of squared deviations is smallest — which is why deviations measured from $\bar{X}$ are systematically smaller than deviations measured from $\mu$.

(b) Unbiasedness of $S^2$. Put $c=\mu$ in the identity and take expectations of both sides, using linearity:

$$E\left[\sum _{i=1}^n(X_i-\bar{X}{)}^2\right]=\sum _{i=1}^{n}E\left[(X_i-\mu {)}^2\right]-nE\left[(\bar{X}-\mu {)}^2\right].$$

Each $E[(X_i-\mu {)}^2]=\mathrm{Var}(X_i)={\sigma }^2$, since every $X_i$ has the population distribution; that gives $n{\sigma }^2$. And because $E(\bar{X})=\mu$, the quantity $E[(\bar{X}-\mu {)}^2]$ is $\mathrm{Var}(\bar{X})={\sigma }^2/n$ — this is where independence enters, through the variance formula. Hence

$$E\left[\sum _{i=1}^n(X_i-\bar{X}{)}^2\right]=n{\sigma }^2-n\cdot \frac{{\sigma }^2}{n}=(n-1){\sigma }^2,E(S^2)=\frac{(n-1){\sigma }^2}{n-1}={\sigma }^2.■$$

For the divide-by-$n$ estimator,

$$E({\hat{\sigma }}^2)=\frac{(n-1){\sigma }^2}{n}={\sigma }^2-\frac{{\sigma }^2}{n},$$

biased low by exactly ${\sigma }^2/n$. The missing piece is $\mathrm{Var}(\bar{X})$: it is the expected squared error incurred by measuring deviations from $\bar{X}$ instead of from $\mu$ — one observation's worth of variance, spent on estimating the mean.

(c) $S$ is biased for $\sigma$. First, $E(S)$ is finite: $S\ge 0$ and $s\le 1+s^2$ for every $s\ge 0$, so $E(S)\le 1+E(S^2)=1+{\sigma }^2<\infty$. So $S$ has a well-defined variance, and by (b),

$$0\le \mathrm{Var}(S)=E(S^2)-[E(S){]}^2={\sigma }^2-[E(S){]}^2.$$

Hence $[E(S){]}^2\le {\sigma }^2$, and since $E(S)\ge 0$ and $\sigma \ge 0$, taking square roots gives $E(S)\le \sigma$. Equality holds if and only if $\mathrm{Var}(S)=0$, which is if and only if $S$ equals a constant with probability $1$. $■$

For any population in which $S$ actually varies — which is every population that takes more than one value — the inequality is strict: the sample standard deviation underestimates $\sigma$ on average, even though its square is unbiased. Nothing about concavity or Jensen's inequality was needed; the non-negativity of a variance already contains the whole fact.

(a) Markov, and a case of equality. Markov's inequality with $a=20$ gives

$$P(Y\ge 20)\le \frac{E(Y)}{20}=\frac{4}{20}=\frac{1}{5}.$$

To attain it, let $Y$ take only the values $0$ and $20$, with $P(Y=20)=\frac{1}{5}$ and $P(Y=0)=\frac{4}{5}$. Then $E(Y)=20\cdot \frac{1}{5}+0\cdot \frac{4}{5}=4$ as required, and $P(Y\ge 20)=\frac{1}{5}$ — equality.

Why this is the equality case: Markov's proof rests on the pointwise inequality $Y\ge a1\{Y\ge a\}$, and that is an equality precisely when $Y$ is $0$ whenever it is below $a$ and exactly $a$ whenever it is at or above $a$. Since a distribution with $E(Y)=4$ achieves the bound, no inequality of the form $P(Y\ge a)\le C\cdot E(Y)/a$ with $C<1$ can hold for all non-negative $Y$: Markov's constant cannot be improved without adding hypotheses.

(b) Chebyshev applied to the sample mean. Two earlier results are needed. First, the theorem on the mean and variance of the sample mean: for i.i.d. observations $E({\bar{X}}_n)=\mu$ and $\mathrm{Var}({\bar{X}}_n)={\sigma }^2/n=1/n$. Second, Chebyshev's inequality applied to the random variable ${\bar{X}}_n$ with $k=0.25$:

$$P(|{\bar{X}}_n-\mu |\ge 0.25)\le \frac{\mathrm{Var}({\bar{X}}_n)}{(0.25{)}^2}=\frac{1/n}{1/16}=\frac{16}{n}.$$

Requiring $\frac{16}{n}\le \frac{1}{100}$ gives $n\ge 1600$; the smallest such $n$ is $1600$. This is a guarantee valid for every population with $\sigma =1$, whatever its shape. (A normal approximation would ask for only about a hundred observations; the difference is the price of assuming nothing.)

(c) Chebyshev's constant is also sharp. Take the three-point distribution

$$X=\{\begin{array}{ll}\mu -2\sigma & \text{with probability }\frac{1}{8},\\ \mu & \text{with probability }\frac{3}{4},\\ \mu +2\sigma & \text{with probability }\frac{1}{8}.\end{array}$$

Mean: the distribution is symmetric about $\mu$, so $E(X)=\mu$; explicitly, $E(X-\mu )=\frac{1}{8}(-2\sigma )+\frac{3}{4}(0)+\frac{1}{8}(2\sigma )=0$.

Variance: $E[(X-\mu {)}^2]=\frac{1}{8}(4{\sigma }^2)+\frac{3}{4}(0)+\frac{1}{8}(4{\sigma }^2)=\frac{{\sigma }^2}{2}+\frac{{\sigma }^2}{2}={\sigma }^2$. ✓

The tail: $P(|X-\mu |\ge 2\sigma )=\frac{1}{8}+\frac{1}{8}=\frac{1}{4}=\frac{1}{2^2}$, exactly Chebyshev's bound with $k=2\sigma$.

Chebyshev's inequality must hold for this distribution as well as for the normal, so the constant $\frac{1}{k^2}$ cannot be lowered. The normal distribution keeps most of its mass close to $\mu$ and puts only about $0.046$ beyond $2\sigma$ — a factor of more than five below the bound. The bound is loose for the normal precisely because it is tight for the worst case: universality is paid for in slack.

(a) Closed forms. For $n\le 4$ every flip is a head: $H_n=n$, ${\hat{p}}_n=1$, and the excess is $n-\frac{n}{2}=\frac{n}{2}$, i.e. $\frac{1}{2},1,\frac{3}{2},2$.

From flip $5$ onward the flips come in pairs $(T,H)$, each pair contributing exactly one head.

Even $n=2j$ with $j\ge 2$. Flips $5,\dots ,2j$ form $j-2$ complete pairs, so

$$H_n=4+(j-2)=j+2,{\hat{p}}_n=\frac{j+2}{2j}=\frac{1}{2}+\frac{1}{j}=\frac{1}{2}+\frac{2}{n},H_n-\frac{n}{2}=(j+2)-j=2.$$

Odd $n=2j+1$ with $j\ge 2$. One more flip, a tail, so $H_n=j+2$ still, and

$${\hat{p}}_n=\frac{j+2}{2j+1},H_n-\frac{n}{2}=(j+2)-\frac{2j+1}{2}=\frac{3}{2},{\hat{p}}_n-\frac{1}{2}=\frac{2(j+2)-(2j+1)}{2(2j+1)}=\frac{3}{2(2j+1)}=\frac{3}{2n}.$$

In both cases ${\hat{p}}_n-\frac{1}{2}=\frac{H_n-n/2}{n}$ — which is in fact an identity for every $n$, since $\frac{H_n}{n}-\frac{1}{2}=\frac{H_n-n/2}{n}$. The excess is $2$ at even $n$ and $\frac{3}{2}$ at odd $n$, for ever.

(b) The sketch of ${\hat{p}}_n$, and the band. The drawing must show: four dots at height $1$ for $n=1,2,3,4$; then a sawtooth that descends towards $0.5$ from above and never crosses it, since ${\hat{p}}_n-\frac{1}{2}$ is always positive. The even-$n$ points lie on the upper envelope $\frac{1}{2}+\frac{2}{n}$ and the odd-$n$ points on the lower envelope $\frac{1}{2}+\frac{3}{2n}$; some values to plot are ${\hat{p}}_5=\frac{4}{5}=0.8$, ${\hat{p}}_6=\frac{5}{6}$, ${\hat{p}}_8=\frac{3}{4}$, ${\hat{p}}_{10}=0.7$, ${\hat{p}}_{12}=\frac{2}{3}$, ${\hat{p}}_{20}=0.6$, ${\hat{p}}_{29}=\frac{16}{29}\approx 0.55$ and ${\hat{p}}_{30}=\frac{17}{30}\approx 0.57$. The teeth shrink as $n$ grows. Draw the three horizontal lines; the line $y=0.4$ is never approached, because the deviation is one-sided.

Entering the band. Strictly inside means $0<{\hat{p}}_n-\frac{1}{2}<\frac{1}{10}$. For even $n$ this is $\frac{2}{n}<\frac{1}{10}$, i.e. $n>20$, so even $n\ge 22$ are inside and even $n\le 20$ are not: ${\hat{p}}_{16}=\frac{10}{16}=0.625$, ${\hat{p}}_{18}=\frac{11}{18}\approx 0.611$, and ${\hat{p}}_{20}=\frac{12}{20}=0.6$ sits exactly on the boundary. For odd $n$ it is $\frac{3}{2n}<\frac{1}{10}$, i.e. $n>15$, so odd $n\ge 17$ are inside: ${\hat{p}}_{17}=\frac{10}{17}\approx 0.588$ and ${\hat{p}}_{19}=\frac{11}{19}\approx 0.579$. The sequence therefore dips into the band at $17$, leaves at $18$, re-enters at $19$, touches the edge at $20$, and the last $n$ not strictly inside is $n=20$. Mark $17$–$20$ on the sketch: this in-and-out is exactly the behaviour the weak law permits.

Proof that it stays inside. For every $n\ge 21$, whichever parity, $0<{\hat{p}}_n-\frac{1}{2}\le \frac{2}{n}\le \frac{2}{21}<\frac{1}{10}$, using $\frac{3}{2n}<\frac{2}{n}$ for the odd case. So ${\hat{p}}_n\in (0.5,0.6)\subset (0.4,0.6)$ for all $n\ge 21$. $■$

(c) The sketch of the excess, and the moral. The drawing shows points at $\frac{1}{2},1,\frac{3}{2},2$ for $n=1,\dots ,4$, and then a flat zigzag between $\frac{3}{2}$ (odd $n$) and $2$ (even $n$) all the way to $n=30$ and beyond. It never approaches the axis $y=0$; in fact it never drops below $\frac{3}{2}$ again.

Why the two pictures differ: after the fourth flip a surplus of two heads exists, and nothing in the later flips ever pays it back — each $(T,H)$ pair has zero net effect on the excess. Yet ${\hat{p}}_n-\frac{1}{2}=\frac{\text{excess}}{n}$, a bounded numerator over a growing denominator, so the proportion converges to $\frac{1}{2}$ all the same. The proportion returns to $\frac{1}{2}$ by dilution, not by compensation.

The Law of Large Numbers is a statement about the proportion — the sample mean of the $0/1$ indicators — and nothing else. It says nothing about the count. For a genuinely random coin the situation is worse than in this hand-made sequence: the excess is not even bounded, its standard deviation being $\frac{1}{2}\sqrt{n}$, which grows without limit. Anyone expecting a run of heads to be "corrected" by later tails is expecting the count to converge, and it never does.

(a) Standardising. A single flip has $\mu =E(X_i)=1\cdot \frac{1}{2}+0\cdot \frac{1}{2}=\frac{1}{2}$ and $E(X_i^2)=\frac{1}{2}$, so ${\sigma }^2=\frac{1}{2}-\frac{1}{4}=\frac{1}{4}$ and $\sigma =\frac{1}{2}$.

The standardised mean is

$$\frac{{\bar{X}}_n-\frac{1}{2}}{\frac{1}{2}/\sqrt{n}}=2\sqrt{n}\left(\frac{H_n}{n}-\frac{1}{2}\right)=\frac{2}{\sqrt{n}}\left(H_n-\frac{n}{2}\right)=\frac{H_n-n/2}{\sqrt{n}/2}=Z_n,$$

and the standardised sum is $\frac{H_n-n\mu }{\sigma \sqrt{n}}=\frac{H_n-n/2}{\sqrt{n}/2}$, the same expression: mean and sum standardise to one and the same variable.

With $Y_i=2X_i-1$: heads gives $Y_i=1$, tails gives $Y_i=-1$, each with probability $\frac{1}{2}$, and the $Y_i$ are independent because the $X_i$ are. Summing, ${\sum }_i{Y}_i=2H_n-n=2\left(H_n-\frac{n}{2}\right)$, so

$$\frac{1}{\sqrt{n}}\sum _{i=1}^n{Y}_i=\frac{2(H_n-n/2)}{\sqrt{n}}=Z_n.✓$$

(b) The MGF. $m(t)=E\left(e^{tY}\right)=\frac{1}{2}{e}^t+\frac{1}{2}{e}^{-t}=\cosh t$. Since $Z_n={\sum }_i(Y_i/\sqrt{n})$ is a sum of independent variables, its MGF is the product of theirs, and each factor is $M_{Y_i/\sqrt{n}}(t)=m(t/\sqrt{n})$ by the scaling rule. Hence

$$M_{Z_n}(t)=\prod _{i=1}^n\cosh \left(\frac{t}{\sqrt{n}}\right)={\left[\cosh \left(\frac{t}{\sqrt{n}}\right)\right]}^n.$$

(c) The limit, with the remainder controlled. Fix $t$ and write $u=t/\sqrt{n}$, so $|u|\le |t|$ for all $n\ge 1$. From the series,

$$\cosh u=1+\frac{u^2}{2}+R(u),R(u)=\sum _{k\ge 2}\frac{u^{2k}}{(2k)!}\ge 0.$$

For $k\ge 2$, $(2k)!=(2k)(2k-1)(2k-2)(2k-3)\cdot (2k-4)!\ge 4\cdot 3\cdot 2\cdot 1\cdot (2k-4)!=24(2k-4)!$, so

$$R(u)\le \frac{u^4}{24}\sum _{k\ge 2}\frac{u^{2k-4}}{(2k-4)!}=\frac{u^4}{24}\sum _{m\ge 0}\frac{u^{2m}}{(2m)!}=\frac{u^4}{24}\cosh u\le \frac{u^4}{24}\cosh t.$$