Print this page and work through it on paper. Revealed answers are never printed, so you can check one question and still print a clean sheet.

1. 1.Computation · 7 min

   The pair $(X,Y)$ has the joint pmf

   $$\begin{array}{cccc}& X=0& X=1& X=2\\ Y=0& \frac{1}{8}& \frac{1}{4}& 0\\ Y=1& \frac{1}{8}& \frac{1}{8}& \frac{3}{8}\end{array}$$

   1. (a)Find the marginal pmf of $X$, the marginal pmf of $Y$, and $E[X]$ and $E[Y]$.
   2. (b)Compute $\mathrm{Cov}(X,Y)$ from the computational formula. Then confirm its sign by deciding which cells of the table contribute positively to $E[(X-{\mu }_X)(Y-{\mu }_Y)]$ and which negatively, and by adding up those weighted contributions.
   3. (c)Compute $\mathrm{Var}(X+Y)$ and $\mathrm{Var}(X-Y)$ without finding the distribution of either variable. Then find the pmf of $X-Y$ directly from the table and verify your value of $\mathrm{Var}(X-Y)$.
   4. (d)Without any further computation, write down $\mathrm{Cov}(2X+1,3-Y)$, naming the property you use.
   Show the solution to question 1

   (a) Marginals by summing the table. Column sums give the pmf of $X$: $P(X=0)=\frac{1}{8}+\frac{1}{8}=\frac{1}{4}$, $P(X=1)=\frac{1}{4}+\frac{1}{8}=\frac{3}{8}$, $P(X=2)=0+\frac{3}{8}=\frac{3}{8}$. Row sums give the pmf of $Y$: $P(Y=0)=\frac{1}{8}+\frac{1}{4}+0=\frac{3}{8}$ and $P(Y=1)=\frac{1}{8}+\frac{1}{8}+\frac{3}{8}=\frac{5}{8}$. Hence

   $$E[X]=0\cdot \frac{1}{4}+1\cdot \frac{3}{8}+2\cdot \frac{3}{8}=\frac{9}{8},E[Y]=\frac{5}{8}.$$

   (b) Computational formula. Only cells with both coordinates nonzero contribute to $E[XY]$: the cell $(1,1)$ gives $1\cdot \frac{1}{8}$ and the cell $(2,1)$ gives $2\cdot \frac{3}{8}$, so $E[XY]=\frac{1}{8}+\frac{6}{8}=\frac{7}{8}$. By the computational formula for the covariance,

   $$\mathrm{Cov}(X,Y)=E[XY]-E[X]E[Y]=\frac{7}{8}-\frac{9}{8}\cdot \frac{5}{8}=\frac{56}{64}-\frac{45}{64}=\frac{11}{64}.$$

   For the sign check, the deviations are $X-\frac{9}{8}\in \{-\frac{9}{8},-\frac{1}{8},\frac{7}{8}\}$ and $Y-\frac{5}{8}\in \{-\frac{5}{8},\frac{3}{8}\}$. In the cells $(0,0)$, $(1,0)$ and $(2,1)$ the two deviations have the same sign, so those cells push the covariance up; in $(0,1)$ and $(1,1)$ they have opposite signs and push it down; the cell $(2,0)$ has probability zero and contributes nothing. Weighting each product of deviations by its cell probability,

   $$\frac{1}{8}\cdot \frac{45}{64}+\frac{1}{4}\cdot \frac{5}{64}+\frac{3}{8}\cdot \frac{21}{64}-\frac{1}{8}\cdot \frac{27}{64}-\frac{1}{8}\cdot \frac{3}{64}=\frac{45+10+63-27-3}{512}=\frac{88}{512}=\frac{11}{64},$$

   the same number from the definition. It is positive because the two heaviest cells, $(2,1)$ and $(1,0)$ together with $(0,0)$, all sit in the agreeing quadrants.

   (c) Variance of a sum, no hypotheses. $E[X^2]=1\cdot \frac{3}{8}+4\cdot \frac{3}{8}=\frac{15}{8}$, so $\mathrm{Var}(X)=\frac{15}{8}-\frac{81}{64}=\frac{120-81}{64}=\frac{39}{64}$. $Y$ is Bernoulli with $p=\frac{5}{8}$, so $\mathrm{Var}(Y)=\frac{5}{8}\cdot \frac{3}{8}=\frac{15}{64}$. By the theorem on the variance of a sum, which needs no independence,

   $$\mathrm{Var}(X+Y)=\frac{39}{64}+\frac{15}{64}+2\cdot \frac{11}{64}=\frac{76}{64}=\frac{19}{16},\mathrm{Var}(X-Y)=\frac{39}{64}+\frac{15}{64}-2\cdot \frac{11}{64}=\frac{32}{64}=\frac{1}{2}.$$

   Direct check: reading $X-Y$ off each cell, it equals $-1$ on $(0,1)$ [probability $\frac{1}{8}$], $0$ on $(0,0)$ and $(1,1)$ [probability $\frac{1}{8}+\frac{1}{8}=\frac{1}{4}$], and $1$ on $(1,0)$ and $(2,1)$ [probability $\frac{1}{4}+\frac{3}{8}=\frac{5}{8}$]. So $E[X-Y]=-\frac{1}{8}+\frac{5}{8}=\frac{1}{2}$, which agrees with $\frac{9}{8}-\frac{5}{8}$, and $E[(X-Y{)}^2]=\frac{1}{8}+\frac{5}{8}=\frac{3}{4}$, giving $\mathrm{Var}(X-Y)=\frac{3}{4}-\frac{1}{4}=\frac{1}{2}$. ✓

   (d) Bilinearity. By the theorem on bilinearity and symmetry of covariance, additive shifts are invisible and scale factors come out of each slot: $\mathrm{Cov}(2X+1,3-Y)=2\cdot (-1)\cdot \mathrm{Cov}(X,Y)=-\frac{11}{32}$.

2. 2.Proof · 8 min

   Throughout, every variable has finite variance. You may use only the definition of covariance, the theorem on bilinearity and symmetry of covariance, the identity $\mathrm{Cov}(X,X)=\mathrm{Var}(X)$, and the variance-of-a-sum theorem. Independence is never assumed unless stated.

   1. (a)Prove that $\mathrm{Cov}(X+Y,X-Y)=\mathrm{Var}(X)-\mathrm{Var}(Y)$. Deduce that if $X$ and $Y$ have the same variance then $X+Y$ and $X-Y$ are uncorrelated, whether or not $X$ and $Y$ are independent.
   2. (b)Let $X_1,\dots ,X_n$ be pairwise uncorrelated, each with variance ${\sigma }^2$, and let $\bar{X}=\frac{1}{n}{\sum }_{i=1}^n{X}_i$. Prove that $\mathrm{Cov}(X_i,\bar{X})={\sigma }^2/n$ for every $i$, and that $\mathrm{Var}(\bar{X})={\sigma }^2/n$.
   3. (c)Deduce that each residual $X_i-\bar{X}$ is uncorrelated with $\bar{X}$, and compute $\mathrm{Var}(X_i-\bar{X})$.
   Show the solution to question 2

   (a) Expand by bilinearity. Linearity in the first slot, then in the second, gives

   $$\mathrm{Cov}(X+Y,X-Y)=\mathrm{Cov}(X,X)-\mathrm{Cov}(X,Y)+\mathrm{Cov}(Y,X)-\mathrm{Cov}(Y,Y).$$

   By symmetry $\mathrm{Cov}(Y,X)=\mathrm{Cov}(X,Y)$, so the two middle terms cancel, and the outer terms are $\mathrm{Var}(X)$ and $\mathrm{Var}(Y)$ because variance is a self-covariance. Hence $\mathrm{Cov}(X+Y,X-Y)=\mathrm{Var}(X)-\mathrm{Var}(Y)$. $■$

   If $\mathrm{Var}(X)=\mathrm{Var}(Y)$ the right-hand side is $0$, which is the definition of $X+Y$ and $X-Y$ being uncorrelated. Nothing in the argument mentioned $\mathrm{Cov}(X,Y)$ itself — it cancelled — so the conclusion holds however strongly $X$ and $Y$ are related. (Uncorrelated is not the same as independent: for $X$, $Y$ independent with equal variances, $X+Y$ and $X-Y$ are uncorrelated but $|X-Y|$ and $X+Y$ are typically dependent.)

   (b) One row of the covariance matrix at a time. Fix $i$. By linearity in the second slot,

   $$\mathrm{Cov}(X_i,\bar{X})=\frac{1}{n}\sum _{j=1}^n\mathrm{Cov}(X_i,X_j)=\frac{1}{n}(\mathrm{Var}(X_i)+\sum _{j\ne i}\mathrm{Cov}(X_i,X_j))=\frac{1}{n}({\sigma }^2+0)=\frac{{\sigma }^2}{n},$$

   since the $j\ne i$ terms vanish by pairwise uncorrelatedness. For the variance, use $\mathrm{Var}(\bar{X})=\mathrm{Cov}(\bar{X},\bar{X})$ and linearity in the first slot:

   $$\mathrm{Var}(\bar{X})=\frac{1}{n}\sum _{i=1}^n\mathrm{Cov}(X_i,\bar{X})=\frac{1}{n}\cdot n\cdot \frac{{\sigma }^2}{n}=\frac{{\sigma }^2}{n}.$$

   (Equivalently, the corollary on the variance of a linear combination with all $a_i=\frac{1}{n}$ and no cross terms gives $n\cdot \frac{1}{n^2}{\sigma }^2$.) $■$

   (c) Residual against mean. By bilinearity and (b),

   $$\mathrm{Cov}(X_i-\bar{X},\bar{X})=\mathrm{Cov}(X_i,\bar{X})-\mathrm{Cov}(\bar{X},\bar{X})=\frac{{\sigma }^2}{n}-\frac{{\sigma }^2}{n}=0.$$

   For the variance, apply the variance-of-a-sum theorem to $X_i+(-1)\bar{X}$:

   $$\mathrm{Var}(X_i-\bar{X})=\mathrm{Var}(X_i)+\mathrm{Var}(\bar{X})-2\mathrm{Cov}(X_i,\bar{X})={\sigma }^2+\frac{{\sigma }^2}{n}-\frac{2{\sigma }^2}{n}={\sigma }^2(1-\frac{1}{n})=\frac{(n-1){\sigma }^2}{n}.$$

   Each residual is a little less variable than the observations themselves, because $\bar{X}$ has already absorbed a $\frac{1}{n}$ share of $X_i$. Summing the $n$ residual variances gives $(n-1){\sigma }^2$, which is exactly why a sample variance divides the sum of squared residuals by $n-1$ rather than $n$.

3. 3.Computation · 7 min

   Let $X$ and $Y$ be independent with $\mathrm{Var}(X)=1$ and $\mathrm{Var}(Y)=4$, and put $U=X+Y$ and $V=X-Y$.

   1. (a)Compute $\mathrm{Cov}(U,V)$, $\mathrm{Var}(U)$ and $\mathrm{Var}(V)$, and hence $\rho (U,V)$.
   2. (b)Now let $\mathrm{Var}(X)=a$ and $\mathrm{Var}(Y)=b$, both strictly positive, with $X$ and $Y$ still independent. Show that $\rho (U,V)=\frac{a-b}{a+b}$. Explain why this can never equal $\pm 1$ — both directly from the formula and from the equality case of the correlation bound — and say exactly when it equals $0$.
   3. (c)Return to $a=1$, $b=4$. Write down $\rho (U,2-3V)$ and $\rho (-U,-V)$, naming the result you use.
   Show the solution to question 3

   (a) Bilinearity, with the cross term killed by independence. Since $X$ and $Y$ are independent, $\mathrm{Cov}(X,Y)=0$ by the theorem that independence implies zero covariance. Expanding,

   $$\mathrm{Cov}(U,V)=\mathrm{Cov}(X+Y,X-Y)=\mathrm{Var}(X)-\mathrm{Var}(Y)=1-4=-3,$$

   the two $\mathrm{Cov}(X,Y)$ terms cancelling by symmetry regardless. By the variance-of-a-sum theorem, $\mathrm{Var}(U)=1+4+0=5$ and $\mathrm{Var}(V)=1+4-0=5$. Therefore

   $$\rho (U,V)=\frac{-3}{\sqrt{5}\sqrt{5}}=-\frac{3}{5}.$$

   (b) The general formula. The same computation gives $\mathrm{Cov}(U,V)=a-b$ and $\mathrm{Var}(U)=\mathrm{Var}(V)=a+b$, so $\rho (U,V)=\frac{a-b}{\sqrt{a+b}\sqrt{a+b}}=\frac{a-b}{a+b}$.

   Directly: since $a,b>0$ we have $|a-b|<a+b$, so $|\rho |<1$ strictly.

   From the equality case of the correlation bound: $|\rho (U,V)|=1$ would force $V=\alpha +\beta U$ almost surely for constants $\alpha$, $\beta \ne 0$, i.e. $(1-\beta )X=\alpha +(1+\beta )Y$ almost surely. If $\beta \ne 1$ this makes $X$ an affine function of $Y$; but $X$ is independent of $Y$, hence independent of every function of $Y$, hence independent of itself, and a variable independent of itself satisfies $P(X\le t)=P(X\le t{)}^2$ for every $t$ and so is constant — contradicting $a>0$. If $\beta =1$ the relation reads $2Y=-\alpha$, so $Y$ is constant, contradicting $b>0$. Either way an exact linear relation between $U$ and $V$ is impossible, which is what the formula was saying.

   $\rho (U,V)=0$ exactly when $a=b$: the sum and difference of two independent variables are uncorrelated precisely when the variances match.

   (c) Correlation is unit-free. By the proposition that $\rho (aX+b,cY+d)=\mathrm{sign}(ac)\rho (X,Y)$: for $\rho (U,2-3V)$ the scale factors are $1$ and $-3$, so the sign flips and $\rho (U,2-3V)=+\frac{3}{5}$. For $\rho (-U,-V)$ the factors are $-1$ and $-1$, whose product is positive, so $\rho (-U,-V)=-\frac{3}{5}$, unchanged.

4. 4.Computation · 8 min

   Let $X$ be uniform on the four values $\{-2,-1,1,2\}$ and let $Y=|X|$.

   1. (a)Compute $E[X]$, $E[Y]$, $E[XY]$ and hence $\mathrm{Cov}(X,Y)$.
   2. (b)Show that $X$ and $Y$ are not independent by exhibiting one pair of events for which the product rule fails.
   3. (c)Compute $\mathrm{Var}(X+Y)$ in two ways: from the variance-of-a-sum identity, and directly from the pmf of $X+Y$. State the hypothesis that licenses dropping the cross term, and say whether it is independence.
   4. (d)Compute $E[X^{2}Y]$ and $E[X^2]E[Y]$. Are they equal? Explain what the comparison shows about which factorisation rules are available for an uncorrelated pair.
   Show the solution to question 4

   (a) Odd symmetry kills the cross moment. The distribution of $X$ is symmetric about $0$, so $E[X]=\frac{1}{4}(-2-1+1+2)=0$. Next $Y=|X|$ takes the value $2$ with probability $\frac{1}{2}$ and $1$ with probability $\frac{1}{2}$, so $E[Y]=\frac{3}{2}$. The product $XY=X|X|$ takes the values $-4,-1,1,4$ with probability $\frac{1}{4}$ each, so $E[XY]=0$. By the computational formula,

   $$\mathrm{Cov}(X,Y)=E[XY]-E[X]E[Y]=0-0\cdot \frac{3}{2}=0.$$

   $X$ and $Y$ are uncorrelated.

   (b) One failing pair of events suffices. Take $\{X=1\}$ and $\{Y=2\}$. They cannot both happen, since $X=1$ forces $Y=1$, so $P(X=1,Y=2)=0$. But $P(X=1)P(Y=2)=\frac{1}{4}\cdot \frac{1}{2}=\frac{1}{8}\ne 0$. The product rule fails, so $X$ and $Y$ are not independent. (Indeed $Y$ is a function of $X$: they are as dependent as possible, and still uncorrelated.)

   (c) Two routes to $\mathrm{Var}(X+Y)$. $\mathrm{Var}(X)=E[X^2]-0=\frac{1}{4}(4+1+1+4)=\frac{5}{2}$. $E[Y^2]=\frac{1}{2}(4)+\frac{1}{2}(1)=\frac{5}{2}$, so $\mathrm{Var}(Y)=\frac{5}{2}-\frac{9}{4}=\frac{1}{4}$. By the variance-of-a-sum theorem,

   $$\mathrm{Var}(X+Y)=\frac{5}{2}+\frac{1}{4}+2\cdot 0=\frac{11}{4}.$$

   Directly: $X+Y=X+|X|$ equals $0$ when $X=-2$ or $X=-1$, equals $2$ when $X=1$, and equals $4$ when $X=2$. So $X+Y$ is $0$ with probability $\frac{1}{2}$, $2$ with probability $\frac{1}{4}$, $4$ with probability $\frac{1}{4}$. Then $E[X+Y]=\frac{1}{2}+1=\frac{3}{2}$ and $E[(X+Y{)}^2]=\frac{1}{4}\cdot 4+\frac{1}{4}\cdot 16=5$, so $\mathrm{Var}(X+Y)=5-\frac{9}{4}=\frac{11}{4}$. ✓

   The hypothesis used to drop the cross term is only $\mathrm{Cov}(X,Y)=0$ — uncorrelatedness. It is not independence, and (b) shows independence is false here; the identity did not need it.

   (d) Where uncorrelatedness stops. $X^{2}Y=|X{|}^2|X|=|X{|}^3$ takes the values $8,1,1,8$ with probability $\frac{1}{4}$ each, so $E[X^{2}Y]=\frac{18}{4}=\frac{9}{2}$. On the other hand $E[X^2]E[Y]=\frac{5}{2}\cdot \frac{3}{2}=\frac{15}{4}$. They are not equal.

   Uncorrelated means exactly one factorisation, $E[XY]=E[X]E[Y]$. Independence would give $E[g(X)h(Y)]=E[g(X)]E[h(Y)]$ for every pair of functions, and here $g(x)=x^2$, $h(y)=y$ already breaks it. So for an uncorrelated pair you may add variances, but you may not factorise expectations of products of arbitrary functions — that requires genuine independence.

5. 5.Computation · 10 min

   Let $(X,Y)$ have joint density $f(x,y)=8xy$ on the triangle $0<x<y<1$, and $0$ elsewhere.

   1. (a)Find the marginal density $f_Y$ and the conditional density $f_{X\mid Y}(x\mid y)$, checking that the latter integrates to $1$. Compute $E[X\mid Y=y]$.
   2. (b)Write down the random variable $E[X\mid Y]$. Compute the number $E[X\mid Y=\frac{1}{2}]$ and the number $\mathrm{Var}(E[X\mid Y])$, and explain why the expression $\mathrm{Var}(E[X\mid Y=\frac{1}{2}])$ is not a meaningful thing to ask for.
   3. (c)Compute $E[XY\mid Y]$ and $E[X^2\mid Y]$, each as an explicit function of $Y$, naming the property that lets $Y$ pass through the conditional expectation.
   4. (d)Find $f_{Y\mid X}(y\mid x)$ and $E[Y\mid X=x]$ in closed form, and check that $E[Y\mid X=x]>x$ for every $x\in (0,1)$.
   Show the solution to question 5

   (a) Slice, then normalise. For $0<y<1$ the marginal is

   $$f_Y(y)={\int }_0^{y}8xydx=8y\cdot \frac{y^2}{2}=4y^3,$$

   which integrates to $1$ over $(0,1)$ as it must. The conditional density is the joint divided by the marginal:

   $$f_{X\mid Y}(x\mid y)=\frac{8xy}{4y^3}=\frac{2x}{y^2},0<x<y,$$

   and ${\int }_0^y\frac{2x}{y^2}dx=\frac{y^2}{y^2}=1$. ✓ Its mean is

   $$E[X\mid Y=y]={\int }_0^{y}x\cdot \frac{2x}{y^2}dx=\frac{2}{y^2}\cdot \frac{y^3}{3}=\frac{2y}{3}.$$

   Given $Y=y$, the variable $X$ has a density increasing linearly across $(0,y)$, so its mean sits two-thirds of the way along — a sensible answer.

   (b) Substitute the random variable for the value. The function $g(y)=\frac{2y}{3}$ evaluated at $Y$ gives the random variable $E[X\mid Y]=\frac{2}{3}Y$. The number $E[X\mid Y=\frac{1}{2}]=g(\frac{1}{2})=\frac{1}{3}$.

   For the variance of $E[X\mid Y]$ we need the moments of $Y$: $E[Y]={\int }_0^{1}4y^{4}dy=\frac{4}{5}$ and $E[Y^2]={\int }_0^{1}4y^{5}dy=\frac{4}{6}=\frac{2}{3}$, so $\mathrm{Var}(Y)=\frac{2}{3}-\frac{16}{25}=\frac{50-48}{75}=\frac{2}{75}$. Hence

   $$\mathrm{Var}(E[X\mid Y])=\mathrm{Var}\left(\frac{2}{3}Y\right)=\frac{4}{9}\cdot \frac{2}{75}=\frac{8}{675}.$$

   $E[X\mid Y=\frac{1}{2}]$ is the single number $\frac{1}{3}$: the mean of $X$ in the sub-population where $Y=\frac{1}{2}$. A number has no distribution, so asking for its variance is a type error (or, if one insists, trivially $0$). Only the random variable $E[X\mid Y]$, which varies with $Y$, has a variance worth computing — and it is $\frac{8}{675}$, not $0$.

   (c) Taking out what is known. Conditional on $Y$, the factor $Y$ is a constant, so by the take-out-what-is-known property $E[XY\mid Y]=YE[X\mid Y]=\frac{2}{3}{Y}^2$. For the second moment, compute in the conditional world:

   $$E[X^2\mid Y=y]={\int }_0^y{x}^2\cdot \frac{2x}{y^2}dx=\frac{2}{y^2}\cdot \frac{y^4}{4}=\frac{y^2}{2},\text{so}E[X^2\mid Y]=\frac{Y^2}{2}.$$

   (d) Condition the other way. The marginal of $X$ is, for $0<x<1$,

   $$f_X(x)={\int }_x^{1}8xydy=4x(1-x^2),$$

   and ${\int }_0^1(4x-4x^3)dx=2-1=1$. ✓ Then

   $$f_{Y\mid X}(y\mid x)=\frac{8xy}{4x(1-x^2)}=\frac{2y}{1-x^2},x<y<1,$$

   and

   $$E[Y\mid X=x]={\int }_x^1\frac{2y^2}{1-x^2}dy=\frac{2}{1-x^2}\cdot \frac{1-x^3}{3}=\frac{2(1+x+x^2)}{3(1+x)},$$

   using $1-x^3=(1-x)(1+x+x^2)$ and $1-x^2=(1-x)(1+x)$. Sanity checks: as $x\to 1$ the value tends to $\frac{2\cdot 3}{3\cdot 2}=1$, and at $x=0$ it is $\frac{2}{3}$, the mean of the density $2y$ on $(0,1)$.

   For the inequality, $E[Y\mid X=x]>x$ is equivalent to $2(1+x+x^2)>3x(1+x)$, i.e. $2-x-x^2>0$, i.e. $(2+x)(1-x)>0$, which holds throughout $(0,1)$. This had to be true: given $X=x$, the variable $Y$ lives on $(x,1)$, so its conditional mean exceeds $x$.

6. 6.Computation · 10 min

   Use the tower property in every part. State clearly what you condition on, and write out the random variable $E[\cdot \mid \cdot ]$ before you average it.

   1. (a)A coin's bias $P$ is equally likely to be $\frac{1}{4}$, $\frac{1}{2}$ or $\frac{3}{4}$. Given $P=p$, the coin is tossed four times independently and $X$ is the number of heads. Find $E[X\mid P]$, $E[X]$ and $P(X=4)$. Explain why $P(X=4)$ is not $(E[P]{)}^4$.
   2. (b)The number $N$ of customers entering a shop in an hour is Poisson with mean $6$. Each customer, independently of $N$ and of the other customers, buys $0$, $1$ or $2$ items with probabilities $\frac{1}{2}$, $\frac{1}{3}$, $\frac{1}{6}$. Find the expected number of items sold in the hour, naming the theorem you use and checking its hypotheses.
   3. (c)A fair coin is tossed repeatedly. Let $T$ be the number of tosses up to and including the first time two consecutive heads appear. By conditioning on the first toss — and, when it is a head, on the second toss as well — set up an equation for $m=E[T]$ and solve it. You may take $E[T]<\infty$ as given.
   Show the solution to question 6

   (a) Condition on the bias. Given $P=p$ the count $X$ is binomial with $n=4$ and success probability $p$, so $E[X\mid P=p]=4p$; substituting the random variable, $E[X\mid P]=4P$. By the tower property,

   $$E[X]=E[E[X\mid P]]=4E[P]=4\cdot \frac{1}{3}(\frac{1}{4}+\frac{1}{2}+\frac{3}{4})=4\cdot \frac{1}{3}\cdot \frac{3}{2}=2.$$

   For the probability, write it as the expectation of an indicator and condition: $P(X=4)=E[P(X=4\mid P)]$, and $P(X=4\mid P=p)=p^4$. So

   $$P(X=4)=E[P^4]=\frac{1}{3}(\frac{1}{256}+\frac{16}{256}+\frac{81}{256})=\frac{1}{3}\cdot \frac{98}{256}=\frac{98}{768}=\frac{49}{384}.$$

   By contrast $(E[P]{)}^4=(\frac{1}{2}{)}^4=\frac{1}{16}=\frac{24}{384}$, about half as large. The tower property averages the conditional probabilities $p^4$; it does not let you average $p$ first and then raise to the fourth power, because $p\mapsto p^4$ is not linear. (It is convex, so the true answer is the larger one — that is Jensen's inequality.)

   (b) Wald's identity. Let $X_i$ be the number of items bought by the $i$-th customer and $S_N=X_1+\cdots +X_N$. The $X_i$ are i.i.d. with finite mean

   $$\mu =0\cdot \frac{1}{2}+1\cdot \frac{1}{3}+2\cdot \frac{1}{6}=\frac{1}{3}+\frac{1}{3}=\frac{2}{3},$$

   $N$ is non-negative integer valued with finite mean $6$, and $N$ is independent of the sequence $(X_i)$. Those are exactly the hypotheses of Wald's identity. Conditioning on $N$: on $\{N=n\}$ the sum has $n$ terms whose distribution is unaffected by the conditioning, so $E[S_N\mid N=n]=n\mu$ and $E[S_N\mid N]=\frac{2}{3}N$. Then

   $$E[S_N]=E\left[\frac{2}{3}N\right]=\frac{2}{3}\cdot 6=4\text{items}.$$

   (c) First-step analysis. Condition on the first toss, and if it is a head, on the second. Three cases cover everything:

   First toss tails (probability $\frac{1}{2}$): one toss has been spent and nothing has been achieved. The tosses that follow are a fresh independent sequence, so the remaining wait has the same distribution as $T$: $E[T\mid \text{T}]=1+m$.

   First toss heads, second tails (probability $\frac{1}{4}$): two tosses spent, and again the process restarts from scratch: $E[T\mid \text{HT}]=2+m$.

   First two tosses heads (probability $\frac{1}{4}$): finished, $T=2$.

   By the tower property (the law of total expectation over these three events),

   $$m=\frac{1}{2}(1+m)+\frac{1}{4}(2+m)+\frac{1}{4}\cdot 2=\frac{1}{2}+\frac{m}{2}+\frac{1}{2}+\frac{m}{4}+\frac{1}{2}=\frac{3}{2}+\frac{3m}{4}.$$

   Hence $\frac{m}{4}=\frac{3}{2}$ and $m=E[T]=6$. The finiteness of $E[T]$ is what permits subtracting $\frac{3m}{4}$ from both sides. Sanity check: a single head takes $2$ tosses on average, and a run of two must take longer; $6$ is plausible.

7. 7.Critique · 9 min

   A course has two sections. A randomly chosen student is in section $A$ with probability $\frac{3}{5}$ and in section $B$ with probability $\frac{2}{5}$. Let $Y$ be the student's section and $X$ their exam score. Within section $A$ the scores have mean $70$ and standard deviation $10$; within section $B$ they have mean $80$ and standard deviation $10$. A student writes:

   Step 1. The mean score is $E[X]=\frac{70+80}{2}=75$.

   Step 2. By the law of total variance, $\mathrm{Var}(X)=E[\mathrm{Var}(X\mid Y)]=\frac{100+100}{2}=100$, since each section has variance $100$.

   Step 3. The other term, $\mathrm{Var}(E[X\mid Y])$, is zero: $E[X\mid Y=A]=70$ is a constant, and a constant has no variance. So the standard deviation of a random student's score is $10$.

   1. (a)Identify every error. For each, say which step it is in, which rule is being misapplied, and whether it happens to change the number in this particular instance.
   2. (b)Compute $E[X]$ and $\mathrm{Var}(X)$ correctly, and state the fraction of $\mathrm{Var}(X)$ that is explained by section membership.
   3. (c)Suppose section $B$'s standard deviation is changed from $10$ to $20$, everything else unchanged. Recompute $\mathrm{Var}(X)$ and say which of the two components moved.
   Show the solution to question 7

   (a) Three errors, one per step.

   Step 1 averages without weights. The tower property says $E[X]={\sum }_{y}E[X\mid Y=y]P(Y=y)$: the group means are weighted by group size. A plain average is right only when the groups are equally likely, and here they are not. This changes the number: $\frac{3}{5}\cdot 70+\frac{2}{5}\cdot 80=42+32=74$, not $75$.

   Step 2 drops a term and again forgets the weights. The law of total variance has two terms, $E[\mathrm{Var}(X\mid Y)]+\mathrm{Var}(E[X\mid Y])$; writing only the first is not the law. Also, the within term is itself a weighted average, $\frac{3}{5}\cdot 100+\frac{2}{5}\cdot 100$. Because the two conditional variances happen to be equal, the missing weights do not change the number here ($100$ either way) — but the method is wrong and would fail the moment the sections had different spreads, as (c) shows.

   Step 3 is a type error. $E[X\mid Y=A]=70$ is indeed a number, but $\mathrm{Var}(E[X\mid Y])$ is the variance of the random variable $E[X\mid Y]$, which equals $70$ on $\{Y=A\}$ and $80$ on $\{Y=B\}$ — it takes two values and is not constant. Confusing the number $E[X\mid Y=y]$ with the random variable $E[X\mid Y]$ is precisely what makes the between-group term disappear. This changes the number.

   (b) The correct computation. $E[X]=74$ from above.

   Between term: $E[X\mid Y]$ takes $70$ with probability $\frac{3}{5}$ and $80$ with probability $\frac{2}{5}$, with mean $74$, so

   $$\mathrm{Var}(E[X\mid Y])=\frac{3}{5}(70-74{)}^2+\frac{2}{5}(80-74{)}^2=\frac{3}{5}\cdot 16+\frac{2}{5}\cdot 36=\frac{48}{5}+\frac{72}{5}=24.$$

   (For a two-point variable this is $p(1-p)d^2=\frac{3}{5}\cdot \frac{2}{5}\cdot 10^2=24$, a useful check.)

   Within term: $E[\mathrm{Var}(X\mid Y)]=\frac{3}{5}\cdot 100+\frac{2}{5}\cdot 100=100$.

   By the law of total variance, $\mathrm{Var}(X)=100+24=124$, so the standard deviation is $\sqrt{124}=2\sqrt{31}$, a little over $11$ rather than $10$. The fraction explained by section is the between share, $\frac{24}{124}=\frac{6}{31}$.

   (c) Only the within term moves. The conditional means are untouched, so the between term is still $24$. The within term becomes $\frac{3}{5}\cdot 100+\frac{2}{5}\cdot 400=60+160=220$ — and now the unweighted average $\frac{100+400}{2}=250$ would be wrong, which is the error Step 2 was lucky to hide. Hence $\mathrm{Var}(X)=220+24=244$.

8. 8.Computation · 9 min

   Two random-mixture variances, each to be split into its within and between parts.

   1. (a)A coin's bias $P$ is equally likely to be $\frac{1}{4}$, $\frac{1}{2}$ or $\frac{3}{4}$; given $P=p$, the coin is tossed four times and $X$ is the number of heads. Compute $\mathrm{Var}(X)$ by the law of total variance, giving the within and between components separately. Compare with the variance of the number of heads in four tosses of a fair coin, and explain in one sentence where the difference comes from.
   2. (b)Customers arrive at a shop as $N\sim \text{Poisson}(6)$; each buys $0$, $1$ or $2$ items with probabilities $\frac{1}{2}$, $\frac{1}{3}$, $\frac{1}{6}$, independently of $N$ and of each other. Let $S_N$ be the total number of items sold. Compute $\mathrm{Var}(S_N)$ from the random-sum formula, identifying the two components. Then show that for a Poisson $N$ with mean $\lambda$ and any item distribution, $\mathrm{Var}(S_N)=\lambda E[X_1^2]$.
   3. (c)Check your answer to (a) independently by computing $E[X^2]$ with the tower property.
   Show the solution to question 8

   (a) Within plus between. Given $P$, the count is binomial, so $E[X\mid P]=4P$ and $\mathrm{Var}(X\mid P)=4P(1-P)$. The moments of $P$: $E[P]=\frac{1}{3}(\frac{1}{4}+\frac{1}{2}+\frac{3}{4})=\frac{1}{2}$ and $E[P^2]=\frac{1}{3}(\frac{1}{16}+\frac{4}{16}+\frac{9}{16})=\frac{14}{48}=\frac{7}{24}$, so $\mathrm{Var}(P)=\frac{7}{24}-\frac{1}{4}=\frac{1}{24}$.

   Within: $E[\mathrm{Var}(X\mid P)]=E[4P-4P^2]=4(\frac{1}{2}-\frac{7}{24})=4\cdot \frac{5}{24}=\frac{5}{6}.$

   Between: $\mathrm{Var}(E[X\mid P])=\mathrm{Var}(4P)=16\cdot \frac{1}{24}=\frac{2}{3}.$

   By the law of total variance, $\mathrm{Var}(X)=\frac{5}{6}+\frac{2}{3}=\frac{5+4}{6}=\frac{3}{2}$.

   A fair coin tossed four times has variance $4\cdot \frac{1}{2}\cdot \frac{1}{2}=1$. The mixture is more variable by $\frac{1}{2}$: not knowing the bias adds a between-group term of $\frac{2}{3}$ that a fixed coin does not have (it also trims the within term slightly, from $1$ to $\frac{5}{6}$, because $p(1-p)$ is largest at $p=\frac{1}{2}$, but the between term dominates).

   (b) Variance of a random sum. With $X_i$ the items bought by customer $i$: $\mu =E[X_1]=\frac{1}{3}+\frac{2}{6}=\frac{2}{3}$, $E[X_1^2]=\frac{1}{3}+\frac{4}{6}=1$, so ${\sigma }^2=1-\frac{4}{9}=\frac{5}{9}$. For a Poisson, $E[N]=\mathrm{Var}(N)=6$. By the corollary on the variance of a random sum,

   $$\mathrm{Var}(S_N)=E[N]{\sigma }^2+\mathrm{Var}(N){\mu }^2=6\cdot \frac{5}{9}+6\cdot \frac{4}{9}=\frac{30}{9}+\frac{24}{9}=6.$$

   The first term, $\frac{10}{3}$, is the within part (uncertainty about how much each customer buys); the second, $\frac{8}{3}$, is the between part (uncertainty about how many customers there are).

   In general, when $N$ is Poisson with mean $\lambda$ the two coefficients coincide, $E[N]=\mathrm{Var}(N)=\lambda$, so

   $$\mathrm{Var}(S_N)=\lambda {\sigma }^2+\lambda {\mu }^2=\lambda ({\sigma }^2+{\mu }^2)=\lambda E[X_1^2],$$

   by the computational formula for the variance. Here $\lambda E[X_1^2]=6\cdot 1=6$. ✓

   (c) Direct check via $E[X^2]$. In the conditional world, $E[X^2\mid P]=\mathrm{Var}(X\mid P)+(E[X\mid P]{)}^2=4P(1-P)+16P^2=4P+12P^2$. By the tower property,

   $$E[X^2]=4E[P]+12E[P^2]=4\cdot \frac{1}{2}+12\cdot \frac{7}{24}=2+\frac{7}{2}=\frac{11}{2}.$$

   Since $E[X]=4E[P]=2$, $\mathrm{Var}(X)=\frac{11}{2}-4=\frac{3}{2}$, agreeing with (a). ✓

9. 9.Computation · 10 min

   $Y$ is equally likely to be $0$, $1$ or $2$. Given $Y=0$, $X=0$. Given $Y=1$, $X$ is equally likely to be $1$ or $3$. Given $Y=2$, $X$ is equally likely to be $0$ or $2$.

   1. (a)Write down the random variable $m(Y)=E[X\mid Y]$ and compute its mean squared error $E[(X-m(Y){)}^2]$ as a predictor of $X$.
   2. (b)Find the best linear predictor $a^{\ast }+b^{\ast }Y$ of $X$ from $Y$, and compute its mean squared error in two ways: from the formula $\mathrm{Var}(X)(1-{\rho }^2)$, and from the decomposition $E[(X-a-bY{)}^2]=E[(X-m(Y){)}^2]+E[(m(Y)-a-bY{)}^2]$.
   3. (c)Find the best constant predictor and its mean squared error, and rank the three predictors.
   4. (d)Compare ${\rho }^2$ with $\mathrm{Var}(m(Y))/\mathrm{Var}(X)$. Say what each fraction measures and why they differ here.
   Show the solution to question 9

   (a) The conditional mean and its error. The group means are $m(0)=0$, $m(1)=\frac{1+3}{2}=2$, $m(2)=\frac{0+2}{2}=1$. So $m(Y)$ takes the values $0$, $2$, $1$ with probability $\frac{1}{3}$ each — note it is not monotone in $Y$.

   By the theorem that conditional expectation minimises mean squared error, its error is $E[\mathrm{Var}(X\mid Y)]$. The conditional variances are $\mathrm{Var}(X\mid Y=0)=0$, and for the two-point groups $\{1,3\}$ and $\{0,2\}$, each symmetric about its mean at distance $1$, $\mathrm{Var}(X\mid Y=1)=\mathrm{Var}(X\mid Y=2)=1$. Hence

   $$E[(X-m(Y){)}^2]=\frac{1}{3}(0+1+1)=\frac{2}{3}.$$

   (b) The best line. First the moments. $E[Y]=1$ and $\mathrm{Var}(Y)=\frac{1}{3}(1+0+1)=\frac{2}{3}$. By the tower property $E[X]=E[m(Y)]=\frac{1}{3}(0+2+1)=1$ and, taking out what is known, $E[XY]=E[Ym(Y)]=\frac{1}{3}(0\cdot 0+1\cdot 2+2\cdot 1)=\frac{4}{3}$, so $\mathrm{Cov}(X,Y)=\frac{4}{3}-1=\frac{1}{3}$. For $\mathrm{Var}(X)$, $E[X^2\mid Y]$ takes the values $0$, $\frac{1+9}{2}=5$, $\frac{0+4}{2}=2$, so $E[X^2]=\frac{1}{3}(0+5+2)=\frac{7}{3}$ and $\mathrm{Var}(X)=\frac{7}{3}-1=\frac{4}{3}$.

   By the proposition on the best linear predictor,

   $$b^{\ast }=\frac{\mathrm{Cov}(X,Y)}{\mathrm{Var}(Y)}=\frac{1/3}{2/3}=\frac{1}{2},a^{\ast }=E[X]-b^{\ast }E[Y]=1-\frac{1}{2}=\frac{1}{2},$$

   so the best line is $\frac{1}{2}+\frac{1}{2}Y$, predicting $\frac{1}{2}$, $1$, $\frac{3}{2}$ at $Y=0,1,2$.

   Via the formula: ${\rho }^2=\frac{\mathrm{Cov}(X,Y{)}^2}{\mathrm{Var}(X)\mathrm{Var}(Y)}=\frac{1/9}{(4/3)(2/3)}=\frac{1/9}{8/9}=\frac{1}{8}$, so the error is $\mathrm{Var}(X)(1-{\rho }^2)=\frac{4}{3}\cdot \frac{7}{8}=\frac{7}{6}$.

   Via the decomposition: the gaps between the line and $m(Y)$ at $Y=0,1,2$ are $\frac{1}{2}-0$, $1-2$, $\frac{3}{2}-1$, with squares $\frac{1}{4}$, $1$, $\frac{1}{4}$ and mean $\frac{1}{3}\cdot \frac{3}{2}=\frac{1}{2}$. Adding the irreducible error from (a), $\frac{2}{3}+\frac{1}{2}=\frac{7}{6}$. ✓ (The cross term vanishes because $X-m(Y)$ is uncorrelated with every function of $Y$, which is what makes the decomposition Pythagorean.)

   (c) The best constant. By the corollary on the best constant predictor, the minimiser is $c=E[X]=1$ with error $\mathrm{Var}(X)=\frac{4}{3}$. Ranking the errors: conditional mean $\frac{2}{3}$ $<$ best line $\frac{7}{6}$ $<$ best constant $\frac{4}{3}$. Each enlargement of the class of allowed predictors can only help, and here each one does.

   (d) Two different fractions of variance explained. ${\rho }^2=\frac{1}{8}$ is the fraction of $\mathrm{Var}(X)$ removed by the best linear predictor: indeed $\frac{4/3-7/6}{4/3}=\frac{1/6}{4/3}=\frac{1}{8}$.

   $\mathrm{Var}(m(Y))=E[m(Y{)}^2]-1=\frac{1}{3}(0+4+1)-1=\frac{2}{3}$, so $\frac{\mathrm{Var}(m(Y))}{\mathrm{Var}(X)}=\frac{2/3}{4/3}=\frac{1}{2}$. By the law of total variance this is the between share — the fraction of $\mathrm{Var}(X)$ removed by the best predictor of any form, since the remaining $\frac{2}{3}$ is exactly the within term.

   They differ because $m(Y)$ is not a linear function of $Y$: it goes $0,2,1$, and a straight line can only capture the tilt of that pattern, not its bend. The gap $\frac{1}{2}-\frac{1}{8}=\frac{3}{8}$ of the variance is the price of insisting on a line. Had $m$ been linear in $Y$, the two fractions would coincide.

10. 10.Sketch · 10 min

    Let $X$ take the value $1$ with probability $\frac{3}{4}$ and the value $4$ with probability $\frac{1}{4}$, and let $\phi (x)=1/x$ on $(0,\infty )$.

    1. (a)Compute $E[X]$, $\phi (E[X])$ and $E[\phi (X)]$, and state which inequality the comparison illustrates, verifying its hypothesis on $\phi$.
    2. (b)Sketch $y=1/x$ for $\frac{1}{2}\le x\le 5$. Mark the two points of the graph above $x=1$ and $x=4$ and draw the chord joining them. Mark the point of the chord above $x=E[X]$ and the point of the curve above $x=E[X]$, label both with their exact heights, and explain why the chord's height there is exactly $E[\phi (X)]$ and not merely some average. Then draw the supporting line to the curve at $\mu =E[X]$, label its slope, and mark where it crosses the vertical lines $x=1$ and $x=4$.
    3. (c)From the picture: which two marked points would have to coincide for equality in Jensen's inequality, and what does that say about $X$? Then, without a sketch, redo (a) for $\phi (x)=\sqrt{x}$ and say which feature of the picture reverses.
    Show the solution to question 10

    (a) The numbers. $E[X]=\frac{3}{4}\cdot 1+\frac{1}{4}\cdot 4=\frac{3}{4}+1=\frac{7}{4}$, so $\phi (E[X])=\frac{4}{7}$. And $E[\phi (X)]=E[1/X]=\frac{3}{4}\cdot 1+\frac{1}{4}\cdot \frac{1}{4}=\frac{3}{4}+\frac{1}{16}=\frac{13}{16}$.

    Since $\frac{13}{16}=\frac{91}{112}$ and $\frac{4}{7}=\frac{64}{112}$, we have $E[\phi (X)]>\phi (E[X])$, with gap $\frac{27}{112}$. This is Jensen's inequality for a convex $\phi$: on $(0,\infty )$, ${\phi }^{\prime \prime }(x)=2/x^3>0$, so $1/x$ is strictly convex there, and $X$ takes values in $(0,\infty )$ with probability one.

    (b) The drawing, feature by feature. The curve $y=1/x$ is decreasing and bends upward (convex) through $(\frac{1}{2},2)$, $(1,1)$, $(2,\frac{1}{2})$, $(4,\frac{1}{4})$, $(5,\frac{1}{5})$.

    The chord. It joins $(1,1)$ to $(4,\frac{1}{4})$; its slope is $\frac{1/4-1}{3}=-\frac{1}{4}$ and its equation is $y=1-\frac{1}{4}(x-1)=\frac{5-x}{4}$. Because $\phi$ is convex, the chord lies above the curve on $(1,4)$ — draw it so.

    The two marked points above $x=\frac{7}{4}$. On the chord, $y=\frac{5-7/4}{4}=\frac{13}{16}$; on the curve, $y=\frac{4}{7}$. Label them $(\frac{7}{4},\frac{13}{16})$ and $(\frac{7}{4},\frac{4}{7})$; the vertical gap between them is $E[\phi (X)]-\phi (E[X])=\frac{27}{112}$.

    Why the chord point is $E[\phi (X)]$. The point $\frac{3}{4}(1,1)+\frac{1}{4}(4,\frac{1}{4})$ is a weighted average of the two endpoints of the chord, with the weights of the distribution, so it lies on the chord. Its $x$-coordinate is $\frac{3}{4}\cdot 1+\frac{1}{4}\cdot 4=E[X]$ and its $y$-coordinate is $\frac{3}{4}\phi (1)+\frac{1}{4}\phi (4)=E[\phi (X)]$. So the chord point above $E[X]$ is not a generic average — it is the exact point $(E[X],E[\phi (X)])$, sitting one quarter of the way from $(1,1)$ to $(4,\frac{1}{4})$. The curve point below it is $(E[X],\phi (E[X]))$. Jensen's inequality is the statement that the first lies above the second.

    The supporting line at $\mu =\frac{7}{4}$. Since $\phi$ is differentiable the supporting line is the tangent, with slope ${\phi }^{\prime }(\mu )=-1/{\mu }^2=-\frac{16}{49}$:

    $$y=\frac{4}{7}-\frac{16}{49}(x-\frac{7}{4})=\frac{8}{7}-\frac{16}{49}x.$$

    It touches the curve at $(\frac{7}{4},\frac{4}{7})$ and lies below the curve everywhere else. At $x=1$ its height is $\frac{8}{7}-\frac{16}{49}=\frac{40}{49}<1$, and at $x=4$ it is $\frac{8}{7}-\frac{64}{49}=-\frac{8}{49}<\frac{1}{4}$; mark both crossings. (It meets the $x$-axis at $x=\frac{7}{2}$.) A check that reproduces the proof of Jensen: the expected height of the tangent at $X$ is $\frac{3}{4}\cdot \frac{40}{49}+\frac{1}{4}\cdot (-\frac{8}{49})=\frac{30-2}{49}=\frac{4}{7}=\phi (\mu )$, because a line through $(\mu ,\phi (\mu ))$ has expectation $\phi (\mu )$ when $E[X-\mu ]=0$. Since $\phi (X)$ lies above the tangent at both mass points, $E[\phi (X)]\ge \phi (\mu )$ — the picture is the proof.

    (c) Equality, and the concave case. Equality in Jensen would mean the chord point $(E[X],E[\phi (X)])$ coincides with the curve point $(E[X],\phi (E[X]))$. For a strictly convex curve the chord lies strictly above the graph between its endpoints, so the two points can only coincide if the chord has no interior — its endpoints are the same point — i.e. the two mass points of $X$ are equal and $X$ is constant almost surely. That is the equality clause of Jensen's inequality for strictly convex $\phi$.

    For $\phi (x)=\sqrt{x}$: ${\phi }^{\prime \prime }(x)=-\frac{1}{4}{x}^{-3/2}<0$, so $\phi$ is strictly concave and the inequality reverses. Numerically $E[\sqrt{X}]=\frac{3}{4}\cdot 1+\frac{1}{4}\cdot 2=\frac{5}{4}$, while $\sqrt{E[X]}=\sqrt{7/4}=\frac{\sqrt{7}}{2}$; since $(\frac{5}{4}{)}^2=\frac{25}{16}<\frac{28}{16}=\frac{7}{4}$, we have $E[\sqrt{X}]<\sqrt{E[X]}$. ✓ The feature that reverses is the position of the chord: for a concave curve the chord lies below the graph, so the point $(E[X],E[\phi (X)])$ now sits under the curve point rather than over it, and the supporting line sits above the curve.

11. 11.Proof · 10 min

    Let $X$ have finite variance, let $Y$ be any random variable, and write $m(Y)=E[X\mid Y]$. Let $h$ be any function with $E[h(Y{)}^2]<\infty$. Quote the tower property and the properties of conditional expectation by name wherever you use them. (You may take for granted that every expectation written below is finite; that follows from the Cauchy–Schwarz inequality.)

    1. (a)Prove that $E[(X-m(Y))h(Y)]=0$. Deduce that $\mathrm{Cov}(X,h(Y))=\mathrm{Cov}(m(Y),h(Y))$.
    2. (b)Deduce that the best linear predictor of $X$ from $Y$ is the same as the best linear predictor of $m(Y)$ from $Y$, and that, when $\mathrm{Var}(X)$ and $\mathrm{Var}(Y)$ are positive, $\rho (X,Y{)}^2\le \mathrm{Var}(m(Y))/\mathrm{Var}(X)$.
    3. (c)Prove that $\mathrm{Var}(X)=\mathrm{Var}(X-m(Y))+\mathrm{Var}(m(Y))$, and hence that $\mathrm{Var}(m(Y))\le \mathrm{Var}(X)$ with equality if and only if $X=m(Y)$ almost surely. Explain how this identity is the law of total variance.
    Show the solution to question 11

    (a) The prediction error is orthogonal to every function of $Y$. By the tower property,

    $$E[(X-m(Y))h(Y)]=E\left[E\right[(X-m(Y))h(Y)\left|Y\right]].$$

    Inside, $h(Y)$ is a function of $Y$, so by the property taking out what is known it passes through the conditional expectation:

    $$E[(X-m(Y))h(Y)\mid Y]=h(Y)E[X-m(Y)\mid Y]=h(Y)(E[X\mid Y]-m(Y))=h(Y)\cdot 0=0,$$

    where the middle step uses linearity of conditional expectation and, once more, taking out what is known applied to $m(Y)$, itself a function of $Y$. The outer expectation of the zero random variable is $0$. $■$

    For the covariance, bilinearity gives $\mathrm{Cov}(X,h(Y))-\mathrm{Cov}(m(Y),h(Y))=\mathrm{Cov}(X-m(Y),h(Y))$, and by the computational formula

    $$\mathrm{Cov}(X-m(Y),h(Y))=E[(X-m(Y))h(Y)]-E[X-m(Y)]E[h(Y)]=0-0\cdot E[h(Y)]=0,$$

    since the first term vanishes by what was just proved and $E[X-m(Y)]=E[X]-E[m(Y)]=0$ by the tower property. Hence $\mathrm{Cov}(X,h(Y))=\mathrm{Cov}(m(Y),h(Y))$. $■$

    (b) Regression only sees the conditional mean. By the proposition on the best linear predictor, the line predicting $X$ from $Y$ has slope $b^{\ast }=\mathrm{Cov}(X,Y)/\mathrm{Var}(Y)$ and intercept $a^{\ast }=E[X]-b^{\ast }E[Y]$. Take $h(y)=y$ in (a): $\mathrm{Cov}(X,Y)=\mathrm{Cov}(m(Y),Y)$, so the slope is the same for $m(Y)$; and $E[X]=E[m(Y)]$ by the tower property, so the intercept is the same too. The two best lines coincide. $■$

    For the bound, $\rho (X,Y{)}^2=\frac{\mathrm{Cov}(X,Y{)}^2}{\mathrm{Var}(X)\mathrm{Var}(Y)}=\frac{\mathrm{Cov}(m(Y),Y{)}^2}{\mathrm{Var}(X)\mathrm{Var}(Y)}$. If $\mathrm{Var}(m(Y))=0$ then $m(Y)$ is almost surely constant, its covariance with $Y$ is $0$, and the inequality reads $0\le 0$. Otherwise the correlation bound (Cauchy–Schwarz applied to the centred pair $m(Y)-E[m(Y)]$ and $Y-E[Y]$) gives $\mathrm{Cov}(m(Y),Y{)}^2\le \mathrm{Var}(m(Y))\mathrm{Var}(Y)$, and dividing,

    $$\rho (X,Y{)}^2\le \frac{\mathrm{Var}(m(Y))\mathrm{Var}(Y)}{\mathrm{Var}(X)\mathrm{Var}(Y)}=\frac{\mathrm{Var}(m(Y))}{\mathrm{Var}(X)}.■$$

    The variance explained linearly can never exceed the variance explained by conditioning.

    (c) Pythagoras for variances. Write $X=(X-m(Y))+m(Y)$. By the variance-of-a-sum theorem,

    $$\mathrm{Var}(X)=\mathrm{Var}(X-m(Y))+\mathrm{Var}(m(Y))+2\mathrm{Cov}(X-m(Y),m(Y)).$$

    Apply (a) with $h=m$ (legitimate since $E[m(Y{)}^2]<\infty$): the covariance is $E[(X-m(Y))m(Y)]-E[X-m(Y)]E[m(Y)]=0-0=0$. The cross term vanishes and the identity follows. $■$

    Since $\mathrm{Var}(X-m(Y))\ge 0$, we get $\mathrm{Var}(m(Y))\le \mathrm{Var}(X)$. Equality holds if and only if $\mathrm{Var}(X-m(Y))=0$, i.e. $X-m(Y)$ is almost surely equal to a constant; that constant is its mean, which is $0$ by the tower property. So equality holds if and only if $X=m(Y)$ almost surely — $X$ is a function of $Y$, and conditioning on $Y$ leaves nothing unexplained. $■$

    As for the law of total variance: $X-m(Y)$ has mean $0$, so $\mathrm{Var}(X-m(Y))=E[(X-m(Y){)}^2]$, and by the tower property this is $E[E[(X-m(Y){)}^2\mid Y]]$. The inner quantity is, by definition, $\mathrm{Var}(X\mid Y)$ — the variance of $X$ about its conditional mean, computed in the conditional world. So the identity reads $\mathrm{Var}(X)=E[\mathrm{Var}(X\mid Y)]+\mathrm{Var}(E[X\mid Y])$, which is the law of total variance, with the within term revealed as the variance of the best predictor's error and the between term as the variance of the prediction itself.

12. 12.Proof · 10 min

    Quote Jensen's inequality, including its equality clause, wherever you use it, and verify the convexity or concavity of the function each time.

    1. (a)Let $a_1,\dots ,a_n>0$ and let $w_1,\dots ,w_n>0$ with ${\sum }_i{w}_i=1$. Prove the weighted arithmetic–geometric mean inequality ${\prod }_{i=1}^n{a}_i^{w_i}\le {\sum }_{i=1}^n{w}_i{a}_i$, and show that equality holds if and only if all the $a_i$ are equal.
    2. (b)Let $X\ge 0$ and $0<r<s$ with $E[X^s]<\infty$. Prove Lyapunov's inequality $(E[X^r]{)}^{1/r}\le (E[X^s]{)}^{1/s}$. (Apply Jensen to $U=X^r$ and $\phi (u)=u^{s/r}$; explain why $E[U]$ is finite.)
    3. (c)Specialise (b) to $r=1$, $s=2$, and show the resulting inequality is also the Cauchy–Schwarz inequality for a suitable pair of variables. Then, for $X$ equal to $1$ with probability $\frac{3}{4}$ and to $4$ with probability $\frac{1}{4}$, compute $(E[\sqrt{X}]{)}^2$ and $E[X]$ and check that they are ordered as (b) with $r=\frac{1}{2}$, $s=1$ predicts.
    Show the solution to question 12

    (a) AM–GM is Jensen for the logarithm. Let $X$ be the random variable taking the value $a_i$ with probability $w_i$ (if some values coincide, their probabilities add). Then $X>0$ with probability one, $E[X]={\sum }_i{w}_i{a}_i$, and by the law of the unconscious statistician $E[\log X]={\sum }_i{w}_i\log a_i=\log {\prod }_i{a}_i^{w_i}$.

    On $(0,\infty )$, $\phi (x)=\log x$ has ${\phi }^{\prime \prime }(x)=-1/x^2<0$, so it is strictly concave. Jensen's inequality for a concave function gives $E[\log X]\le \log E[X]$, that is,

    $$\log \prod _i{a}_i^{w_i}\le \log \sum _i{w}_i{a}_i.$$

    Since $\exp$ is increasing, ${\prod }_i{a}_i^{w_i}\le {\sum }_i{w}_i{a}_i$. $■$

    By the equality clause for a strictly concave (equivalently, $-\log$ strictly convex) function, equality holds if and only if $X$ is almost surely constant. Every $w_i$ is positive, so each $a_i$ is a value taken with positive probability, and "almost surely constant" means all $a_i$ equal that one constant. Conversely if all $a_i=a$ both sides equal $a$. $■$

    (b) Lyapunov's inequality. Put $U=X^r\ge 0$ and $q=s/r>1$, so that $U^q=X^s$ and $E[U^q]=E[X^s]<\infty$. First, $E[U]$ is finite: for $u\ge 0$ we have $u\le 1+u^q$ (if $u\le 1$ then $u\le 1$; if $u>1$ then $u<u^q$ because $q>1$), so by monotonicity of expectation $E[U]\le 1+E[U^q]<\infty$. Thus $U$ has a finite mean in $[0,\infty )$, and $E[\phi (U)]=E[U^q]$ exists.

    The function $\phi (u)=u^q$ on $[0,\infty )$ is convex: on $(0,\infty )$ its second derivative $q(q-1)u^{q-2}$ is positive because $q>1$, and $\phi$ is continuous at $0$, so the chord condition holds on the whole closed half-line. By Jensen's inequality,

    $$(E[U]{)}^q=\phi (E[U])\le E[\phi (U)]=E[U^q],\text{i.e.}(E[X^r]{)}^{s/r}\le E[X^s].$$

    Both sides are non-negative and $t\mapsto t^{1/s}$ is increasing on $[0,\infty )$, so raising both sides to the power $1/s$ preserves the inequality:

    $$(E[X^r]{)}^{1/r}\le (E[X^s]{)}^{1/s}.■$$

    (As $\phi$ is strictly convex, equality holds if and only if $U$, hence $X$, is almost surely constant.)

    (c) The case $r=1$, $s=2$, and a numerical check. With $r=1$, $s=2$ the inequality reads $E[X]\le (E[X^2]{)}^{1/2}$, i.e. $(E[X]{)}^2\le E[X^2]$ for $X\ge 0$ — the statement that a variance is non-negative. The Cauchy–Schwarz inequality for random variables, applied to the pair $U=X$ and $V=1$, gives

    $$(E[X\cdot 1]{)}^2\le E[X^2]E[1^2]=E[X^2],$$

    the same inequality; and its equality clause — $X$ proportional to the constant $1$ almost surely — is the same as Jensen's: $X$ almost surely constant.

    For the two-point variable, $E[\sqrt{X}]=\frac{3}{4}\cdot 1+\frac{1}{4}\cdot 2=\frac{5}{4}$, so $(E[\sqrt{X}]{)}^2=\frac{25}{16}$, while $E[X]=\frac{3}{4}+1=\frac{7}{4}=\frac{28}{16}$. Lyapunov with $r=\frac{1}{2}$, $s=1$ predicts $(E[X^{1/2}]{)}^2\le E[X]$, and indeed $\frac{25}{16}<\frac{28}{16}$ — strictly, as it must be since $X$ is not constant. ✓