Expectation, Covariance and Conditional Expectation

(a) Marginals by summing the table. Column sums give the pmf of $X$: $P(X=0)=\frac{1}{8}+\frac{1}{8}=\frac{1}{4}$, $P(X=1)=\frac{1}{4}+\frac{1}{8}=\frac{3}{8}$, $P(X=2)=0+\frac{3}{8}=\frac{3}{8}$. Row sums give the pmf of $Y$: $P(Y=0)=\frac{1}{8}+\frac{1}{4}+0=\frac{3}{8}$ and $P(Y=1)=\frac{1}{8}+\frac{1}{8}+\frac{3}{8}=\frac{5}{8}$. Hence

$$E[X]=0\cdot \frac{1}{4}+1\cdot \frac{3}{8}+2\cdot \frac{3}{8}=\frac{9}{8},E[Y]=\frac{5}{8}.$$

(b) Computational formula. Only cells with both coordinates nonzero contribute to $E[XY]$: the cell $(1,1)$ gives $1\cdot \frac{1}{8}$ and the cell $(2,1)$ gives $2\cdot \frac{3}{8}$, so $E[XY]=\frac{1}{8}+\frac{6}{8}=\frac{7}{8}$. By the computational formula for the covariance,

$$\mathrm{Cov}(X,Y)=E[XY]-E[X]E[Y]=\frac{7}{8}-\frac{9}{8}\cdot \frac{5}{8}=\frac{56}{64}-\frac{45}{64}=\frac{11}{64}.$$

For the sign check, the deviations are $X-\frac{9}{8}\in \{-\frac{9}{8},-\frac{1}{8},\frac{7}{8}\}$ and $Y-\frac{5}{8}\in \{-\frac{5}{8},\frac{3}{8}\}$. In the cells $(0,0)$, $(1,0)$ and $(2,1)$ the two deviations have the same sign, so those cells push the covariance up; in $(0,1)$ and $(1,1)$ they have opposite signs and push it down; the cell $(2,0)$ has probability zero and contributes nothing. Weighting each product of deviations by its cell probability,

$$\frac{1}{8}\cdot \frac{45}{64}+\frac{1}{4}\cdot \frac{5}{64}+\frac{3}{8}\cdot \frac{21}{64}-\frac{1}{8}\cdot \frac{27}{64}-\frac{1}{8}\cdot \frac{3}{64}=\frac{45+10+63-27-3}{512}=\frac{88}{512}=\frac{11}{64},$$

the same number from the definition. It is positive because the two heaviest cells, $(2,1)$ and $(1,0)$ together with $(0,0)$, all sit in the agreeing quadrants.

(c) Variance of a sum, no hypotheses. $E[X^2]=1\cdot \frac{3}{8}+4\cdot \frac{3}{8}=\frac{15}{8}$, so $\mathrm{Var}(X)=\frac{15}{8}-\frac{81}{64}=\frac{120-81}{64}=\frac{39}{64}$. $Y$ is Bernoulli with $p=\frac{5}{8}$, so $\mathrm{Var}(Y)=\frac{5}{8}\cdot \frac{3}{8}=\frac{15}{64}$. By the theorem on the variance of a sum, which needs no independence,

$$\mathrm{Var}(X+Y)=\frac{39}{64}+\frac{15}{64}+2\cdot \frac{11}{64}=\frac{76}{64}=\frac{19}{16},\mathrm{Var}(X-Y)=\frac{39}{64}+\frac{15}{64}-2\cdot \frac{11}{64}=\frac{32}{64}=\frac{1}{2}.$$

Direct check: reading $X-Y$ off each cell, it equals $-1$ on $(0,1)$ [probability $\frac{1}{8}$], $0$ on $(0,0)$ and $(1,1)$ [probability $\frac{1}{8}+\frac{1}{8}=\frac{1}{4}$], and $1$ on $(1,0)$ and $(2,1)$ [probability $\frac{1}{4}+\frac{3}{8}=\frac{5}{8}$]. So $E[X-Y]=-\frac{1}{8}+\frac{5}{8}=\frac{1}{2}$, which agrees with $\frac{9}{8}-\frac{5}{8}$, and $E[(X-Y{)}^2]=\frac{1}{8}+\frac{5}{8}=\frac{3}{4}$, giving $\mathrm{Var}(X-Y)=\frac{3}{4}-\frac{1}{4}=\frac{1}{2}$. ✓

(d) Bilinearity. By the theorem on bilinearity and symmetry of covariance, additive shifts are invisible and scale factors come out of each slot: $\mathrm{Cov}(2X+1,3-Y)=2\cdot (-1)\cdot \mathrm{Cov}(X,Y)=-\frac{11}{32}$.

(a) Expand by bilinearity. Linearity in the first slot, then in the second, gives

$$\mathrm{Cov}(X+Y,X-Y)=\mathrm{Cov}(X,X)-\mathrm{Cov}(X,Y)+\mathrm{Cov}(Y,X)-\mathrm{Cov}(Y,Y).$$

By symmetry $\mathrm{Cov}(Y,X)=\mathrm{Cov}(X,Y)$, so the two middle terms cancel, and the outer terms are $\mathrm{Var}(X)$ and $\mathrm{Var}(Y)$ because variance is a self-covariance. Hence $\mathrm{Cov}(X+Y,X-Y)=\mathrm{Var}(X)-\mathrm{Var}(Y)$. $■$

If $\mathrm{Var}(X)=\mathrm{Var}(Y)$ the right-hand side is $0$, which is the definition of $X+Y$ and $X-Y$ being uncorrelated. Nothing in the argument mentioned $\mathrm{Cov}(X,Y)$ itself — it cancelled — so the conclusion holds however strongly $X$ and $Y$ are related. (Uncorrelated is not the same as independent: for $X$, $Y$ independent with equal variances, $X+Y$ and $X-Y$ are uncorrelated but $|X-Y|$ and $X+Y$ are typically dependent.)

(b) One row of the covariance matrix at a time. Fix $i$. By linearity in the second slot,

$$\mathrm{Cov}(X_i,\bar{X})=\frac{1}{n}\sum _{j=1}^n\mathrm{Cov}(X_i,X_j)=\frac{1}{n}(\mathrm{Var}(X_i)+\sum _{j\ne i}\mathrm{Cov}(X_i,X_j))=\frac{1}{n}({\sigma }^2+0)=\frac{{\sigma }^2}{n},$$

since the $j\ne i$ terms vanish by pairwise uncorrelatedness. For the variance, use $\mathrm{Var}(\bar{X})=\mathrm{Cov}(\bar{X},\bar{X})$ and linearity in the first slot:

$$\mathrm{Var}(\bar{X})=\frac{1}{n}\sum _{i=1}^n\mathrm{Cov}(X_i,\bar{X})=\frac{1}{n}\cdot n\cdot \frac{{\sigma }^2}{n}=\frac{{\sigma }^2}{n}.$$

(Equivalently, the corollary on the variance of a linear combination with all $a_i=\frac{1}{n}$ and no cross terms gives $n\cdot \frac{1}{n^2}{\sigma }^2$.) $■$

(c) Residual against mean. By bilinearity and (b),

$$\mathrm{Cov}(X_i-\bar{X},\bar{X})=\mathrm{Cov}(X_i,\bar{X})-\mathrm{Cov}(\bar{X},\bar{X})=\frac{{\sigma }^2}{n}-\frac{{\sigma }^2}{n}=0.$$

For the variance, apply the variance-of-a-sum theorem to $X_i+(-1)\bar{X}$:

$$\mathrm{Var}(X_i-\bar{X})=\mathrm{Var}(X_i)+\mathrm{Var}(\bar{X})-2\mathrm{Cov}(X_i,\bar{X})={\sigma }^2+\frac{{\sigma }^2}{n}-\frac{2{\sigma }^2}{n}={\sigma }^2(1-\frac{1}{n})=\frac{(n-1){\sigma }^2}{n}.$$

Each residual is a little less variable than the observations themselves, because $\bar{X}$ has already absorbed a $\frac{1}{n}$ share of $X_i$. Summing the $n$ residual variances gives $(n-1){\sigma }^2$, which is exactly why a sample variance divides the sum of squared residuals by $n-1$ rather than $n$.

(a) Bilinearity, with the cross term killed by independence. Since $X$ and $Y$ are independent, $\mathrm{Cov}(X,Y)=0$ by the theorem that independence implies zero covariance. Expanding,

$$\mathrm{Cov}(U,V)=\mathrm{Cov}(X+Y,X-Y)=\mathrm{Var}(X)-\mathrm{Var}(Y)=1-4=-3,$$

the two $\mathrm{Cov}(X,Y)$ terms cancelling by symmetry regardless. By the variance-of-a-sum theorem, $\mathrm{Var}(U)=1+4+0=5$ and $\mathrm{Var}(V)=1+4-0=5$. Therefore

$$\rho (U,V)=\frac{-3}{\sqrt{5}\sqrt{5}}=-\frac{3}{5}.$$

(b) The general formula. The same computation gives $\mathrm{Cov}(U,V)=a-b$ and $\mathrm{Var}(U)=\mathrm{Var}(V)=a+b$, so $\rho (U,V)=\frac{a-b}{\sqrt{a+b}\sqrt{a+b}}=\frac{a-b}{a+b}$.

Directly: since $a,b>0$ we have $|a-b|<a+b$, so $|\rho |<1$ strictly.

From the equality case of the correlation bound: $|\rho (U,V)|=1$ would force $V=\alpha +\beta U$ almost surely for constants $\alpha$, $\beta \ne 0$, i.e. $(1-\beta )X=\alpha +(1+\beta )Y$ almost surely. If $\beta \ne 1$ this makes $X$ an affine function of $Y$; but $X$ is independent of $Y$, hence independent of every function of $Y$, hence independent of itself, and a variable independent of itself satisfies $P(X\le t)=P(X\le t{)}^2$ for every $t$ and so is constant — contradicting $a>0$. If $\beta =1$ the relation reads $2Y=-\alpha$, so $Y$ is constant, contradicting $b>0$. Either way an exact linear relation between $U$ and $V$ is impossible, which is what the formula was saying.

$\rho (U,V)=0$ exactly when $a=b$: the sum and difference of two independent variables are uncorrelated precisely when the variances match.

(c) Correlation is unit-free. By the proposition that $\rho (aX+b,cY+d)=\mathrm{sign}(ac)\rho (X,Y)$: for $\rho (U,2-3V)$ the scale factors are $1$ and $-3$, so the sign flips and $\rho (U,2-3V)=+\frac{3}{5}$. For $\rho (-U,-V)$ the factors are $-1$ and $-1$, whose product is positive, so $\rho (-U,-V)=-\frac{3}{5}$, unchanged.

(a) Odd symmetry kills the cross moment. The distribution of $X$ is symmetric about $0$, so $E[X]=\frac{1}{4}(-2-1+1+2)=0$. Next $Y=|X|$ takes the value $2$ with probability $\frac{1}{2}$ and $1$ with probability $\frac{1}{2}$, so $E[Y]=\frac{3}{2}$. The product $XY=X|X|$ takes the values $-4,-1,1,4$ with probability $\frac{1}{4}$ each, so $E[XY]=0$. By the computational formula,

$$\mathrm{Cov}(X,Y)=E[XY]-E[X]E[Y]=0-0\cdot \frac{3}{2}=0.$$

$X$ and $Y$ are uncorrelated.

(b) One failing pair of events suffices. Take $\{X=1\}$ and $\{Y=2\}$. They cannot both happen, since $X=1$ forces $Y=1$, so $P(X=1,Y=2)=0$. But $P(X=1)P(Y=2)=\frac{1}{4}\cdot \frac{1}{2}=\frac{1}{8}\ne 0$. The product rule fails, so $X$ and $Y$ are not independent. (Indeed $Y$ is a function of $X$: they are as dependent as possible, and still uncorrelated.)

(c) Two routes to $\mathrm{Var}(X+Y)$. $\mathrm{Var}(X)=E[X^2]-0=\frac{1}{4}(4+1+1+4)=\frac{5}{2}$. $E[Y^2]=\frac{1}{2}(4)+\frac{1}{2}(1)=\frac{5}{2}$, so $\mathrm{Var}(Y)=\frac{5}{2}-\frac{9}{4}=\frac{1}{4}$. By the variance-of-a-sum theorem,

$$\mathrm{Var}(X+Y)=\frac{5}{2}+\frac{1}{4}+2\cdot 0=\frac{11}{4}.$$

Directly: $X+Y=X+|X|$ equals $0$ when $X=-2$ or $X=-1$, equals $2$ when $X=1$, and equals $4$ when $X=2$. So $X+Y$ is $0$ with probability $\frac{1}{2}$, $2$ with probability $\frac{1}{4}$, $4$ with probability $\frac{1}{4}$. Then $E[X+Y]=\frac{1}{2}+1=\frac{3}{2}$ and $E[(X+Y{)}^2]=\frac{1}{4}\cdot 4+\frac{1}{4}\cdot 16=5$, so $\mathrm{Var}(X+Y)=5-\frac{9}{4}=\frac{11}{4}$. ✓

The hypothesis used to drop the cross term is only $\mathrm{Cov}(X,Y)=0$ — uncorrelatedness. It is not independence, and (b) shows independence is false here; the identity did not need it.

(d) Where uncorrelatedness stops. $X^{2}Y=|X{|}^2|X|=|X{|}^3$ takes the values $8,1,1,8$ with probability $\frac{1}{4}$ each, so $E[X^{2}Y]=\frac{18}{4}=\frac{9}{2}$. On the other hand $E[X^2]E[Y]=\frac{5}{2}\cdot \frac{3}{2}=\frac{15}{4}$. They are not equal.

Uncorrelated means exactly one factorisation, $E[XY]=E[X]E[Y]$. Independence would give $E[g(X)h(Y)]=E[g(X)]E[h(Y)]$ for every pair of functions, and here $g(x)=x^2$, $h(y)=y$ already breaks it. So for an uncorrelated pair you may add variances, but you may not factorise expectations of products of arbitrary functions — that requires genuine independence.

(a) Slice, then normalise. For $0<y<1$ the marginal is

$$f_Y(y)={\int }_0^{y}8xydx=8y\cdot \frac{y^2}{2}=4y^3,$$

which integrates to $1$ over $(0,1)$ as it must. The conditional density is the joint divided by the marginal:

$$f_{X\mid Y}(x\mid y)=\frac{8xy}{4y^3}=\frac{2x}{y^2},0<x<y,$$

and ${\int }_0^y\frac{2x}{y^2}dx=\frac{y^2}{y^2}=1$. ✓ Its mean is

$$E[X\mid Y=y]={\int }_0^{y}x\cdot \frac{2x}{y^2}dx=\frac{2}{y^2}\cdot \frac{y^3}{3}=\frac{2y}{3}.$$

Given $Y=y$, the variable $X$ has a density increasing linearly across $(0,y)$, so its mean sits two-thirds of the way along — a sensible answer.

(b) Substitute the random variable for the value. The function $g(y)=\frac{2y}{3}$ evaluated at $Y$ gives the random variable $E[X\mid Y]=\frac{2}{3}Y$. The number $E[X\mid Y=\frac{1}{2}]=g(\frac{1}{2})=\frac{1}{3}$.

For the variance of $E[X\mid Y]$ we need the moments of $Y$: $E[Y]={\int }_0^{1}4y^{4}dy=\frac{4}{5}$ and $E[Y^2]={\int }_0^{1}4y^{5}dy=\frac{4}{6}=\frac{2}{3}$, so $\mathrm{Var}(Y)=\frac{2}{3}-\frac{16}{25}=\frac{50-48}{75}=\frac{2}{75}$. Hence

$$\mathrm{Var}(E[X\mid Y])=\mathrm{Var}\left(\frac{2}{3}Y\right)=\frac{4}{9}\cdot \frac{2}{75}=\frac{8}{675}.$$

$E[X\mid Y=\frac{1}{2}]$ is the single number $\frac{1}{3}$: the mean of $X$ in the sub-population where $Y=\frac{1}{2}$. A number has no distribution, so asking for its variance is a type error (or, if one insists, trivially $0$). Only the random variable $E[X\mid Y]$, which varies with $Y$, has a variance worth computing — and it is $\frac{8}{675}$, not $0$.

(c) Taking out what is known. Conditional on $Y$, the factor $Y$ is a constant, so by the take-out-what-is-known property $E[XY\mid Y]=YE[X\mid Y]=\frac{2}{3}{Y}^2$. For the second moment, compute in the conditional world:

$$E[X^2\mid Y=y]={\int }_0^y{x}^2\cdot \frac{2x}{y^2}dx=\frac{2}{y^2}\cdot \frac{y^4}{4}=\frac{y^2}{2},\text{so}E[X^2\mid Y]=\frac{Y^2}{2}.$$

(d) Condition the other way. The marginal of $X$ is, for $0<x<1$,

$$f_X(x)={\int }_x^{1}8xydy=4x(1-x^2),$$

and ${\int }_0^1(4x-4x^3)dx=2-1=1$. ✓ Then

$$f_{Y\mid X}(y\mid x)=\frac{8xy}{4x(1-x^2)}=\frac{2y}{1-x^2},x<y<1,$$

and

$$E[Y\mid X=x]={\int }_x^1\frac{2y^2}{1-x^2}dy=\frac{2}{1-x^2}\cdot \frac{1-x^3}{3}=\frac{2(1+x+x^2)}{3(1+x)},$$

using $1-x^3=(1-x)(1+x+x^2)$ and $1-x^2=(1-x)(1+x)$. Sanity checks: as $x\to 1$ the value tends to $\frac{2\cdot 3}{3\cdot 2}=1$, and at $x=0$ it is $\frac{2}{3}$, the mean of the density $2y$ on $(0,1)$.

For the inequality, $E[Y\mid X=x]>x$ is equivalent to $2(1+x+x^2)>3x(1+x)$, i.e. $2-x-x^2>0$, i.e. $(2+x)(1-x)>0$, which holds throughout $(0,1)$. This had to be true: given $X=x$, the variable $Y$ lives on $(x,1)$, so its conditional mean exceeds $x$.

(a) Condition on the bias. Given $P=p$ the count $X$ is binomial with $n=4$ and success probability $p$, so $E[X\mid P=p]=4p$; substituting the random variable, $E[X\mid P]=4P$. By the tower property,

$$E[X]=E[E[X\mid P]]=4E[P]=4\cdot \frac{1}{3}(\frac{1}{4}+\frac{1}{2}+\frac{3}{4})=4\cdot \frac{1}{3}\cdot \frac{3}{2}=2.$$

For the probability, write it as the expectation of an indicator and condition: $P(X=4)=E[P(X=4\mid P)]$, and $P(X=4\mid P=p)=p^4$. So

$$P(X=4)=E[P^4]=\frac{1}{3}(\frac{1}{256}+\frac{16}{256}+\frac{81}{256})=\frac{1}{3}\cdot \frac{98}{256}=\frac{98}{768}=\frac{49}{384}.$$

By contrast $(E[P]{)}^4=(\frac{1}{2}{)}^4=\frac{1}{16}=\frac{24}{384}$, about half as large. The tower property averages the conditional probabilities $p^4$; it does not let you average $p$ first and then raise to the fourth power, because $p\mapsto p^4$ is not linear. (It is convex, so the true answer is the larger one — that is Jensen's inequality.)

(b) Wald's identity. Let $X_i$ be the number of items bought by the $i$-th customer and $S_N=X_1+\cdots +X_N$. The $X_i$ are i.i.d. with finite mean

$$\mu =0\cdot \frac{1}{2}+1\cdot \frac{1}{3}+2\cdot \frac{1}{6}=\frac{1}{3}+\frac{1}{3}=\frac{2}{3},$$

$N$ is non-negative integer valued with finite mean $6$, and $N$ is independent of the sequence $(X_i)$. Those are exactly the hypotheses of Wald's identity. Conditioning on $N$: on $\{N=n\}$ the sum has $n$ terms whose distribution is unaffected by the conditioning, so $E[S_N\mid N=n]=n\mu$ and $E[S_N\mid N]=\frac{2}{3}N$. Then

$$E[S_N]=E\left[\frac{2}{3}N\right]=\frac{2}{3}\cdot 6=4\text{items}.$$

(c) First-step analysis. Condition on the first toss, and if it is a head, on the second. Three cases cover everything:

First toss tails (probability $\frac{1}{2}$): one toss has been spent and nothing has been achieved. The tosses that follow are a fresh independent sequence, so the remaining wait has the same distribution as $T$: $E[T\mid \text{T}]=1+m$.

First toss heads, second tails (probability $\frac{1}{4}$): two tosses spent, and again the process restarts from scratch: $E[T\mid \text{HT}]=2+m$.

First two tosses heads (probability $\frac{1}{4}$): finished, $T=2$.

By the tower property (the law of total expectation over these three events),

$$m=\frac{1}{2}(1+m)+\frac{1}{4}(2+m)+\frac{1}{4}\cdot 2=\frac{1}{2}+\frac{m}{2}+\frac{1}{2}+\frac{m}{4}+\frac{1}{2}=\frac{3}{2}+\frac{3m}{4}.$$

Hence $\frac{m}{4}=\frac{3}{2}$ and $m=E[T]=6$. The finiteness of $E[T]$ is what permits subtracting $\frac{3m}{4}$ from both sides. Sanity check: a single head takes $2$ tosses on average, and a run of two must take longer; $6$ is plausible.

(a) Three errors, one per step.

Step 1 averages without weights. The tower property says $E[X]={\sum }_{y}E[X\mid Y=y]P(Y=y)$: the group means are weighted by group size. A plain average is right only when the groups are equally likely, and here they are not. This changes the number: $\frac{3}{5}\cdot 70+\frac{2}{5}\cdot 80=42+32=74$, not $75$.

Step 2 drops a term and again forgets the weights. The law of total variance has two terms, $E[\mathrm{Var}(X\mid Y)]+\mathrm{Var}(E[X\mid Y])$; writing only the first is not the law. Also, the within term is itself a weighted average, $\frac{3}{5}\cdot 100+\frac{2}{5}\cdot 100$. Because the two conditional variances happen to be equal, the missing weights do not change the number here ($100$ either way) — but the method is wrong and would fail the moment the sections had different spreads, as (c) shows.

Step 3 is a type error. $E[X\mid Y=A]=70$ is indeed a number, but $\mathrm{Var}(E[X\mid Y])$ is the variance of the random variable $E[X\mid Y]$, which equals $70$ on $\{Y=A\}$ and $80$ on $\{Y=B\}$ — it takes two values and is not constant. Confusing the number $E[X\mid Y=y]$ with the random variable $E[X\mid Y]$ is precisely what makes the between-group term disappear. This changes the number.

(b) The correct computation. $E[X]=74$ from above.

Between term: $E[X\mid Y]$ takes $70$ with probability $\frac{3}{5}$ and $80$ with probability $\frac{2}{5}$, with mean $74$, so

$$\mathrm{Var}(E[X\mid Y])=\frac{3}{5}(70-74{)}^2+\frac{2}{5}(80-74{)}^2=\frac{3}{5}\cdot 16+\frac{2}{5}\cdot 36=\frac{48}{5}+\frac{72}{5}=24.$$

(For a two-point variable this is $p(1-p)d^2=\frac{3}{5}\cdot \frac{2}{5}\cdot 10^2=24$, a useful check.)

Within term: $E[\mathrm{Var}(X\mid Y)]=\frac{3}{5}\cdot 100+\frac{2}{5}\cdot 100=100$.

By the law of total variance, $\mathrm{Var}(X)=100+24=124$, so the standard deviation is $\sqrt{124}=2\sqrt{31}$, a little over $11$ rather than $10$. The fraction explained by section is the between share, $\frac{24}{124}=\frac{6}{31}$.

(c) Only the within term moves. The conditional means are untouched, so the between term is still $24$. The within term becomes $\frac{3}{5}\cdot 100+\frac{2}{5}\cdot 400=60+160=220$ — and now the unweighted average $\frac{100+400}{2}=250$ would be wrong, which is the error Step 2 was lucky to hide. Hence $\mathrm{Var}(X)=220+24=244$.

(a) Within plus between. Given $P$, the count is binomial, so $E[X\mid P]=4P$ and $\mathrm{Var}(X\mid P)=4P(1-P)$. The moments of $P$: $E[P]=\frac{1}{3}(\frac{1}{4}+\frac{1}{2}+\frac{3}{4})=\frac{1}{2}$ and $E[P^2]=\frac{1}{3}(\frac{1}{16}+\frac{4}{16}+\frac{9}{16})=\frac{14}{48}=\frac{7}{24}$, so $\mathrm{Var}(P)=\frac{7}{24}-\frac{1}{4}=\frac{1}{24}$.

Within: $E[\mathrm{Var}(X\mid P)]=E[4P-4P^2]=4(\frac{1}{2}-\frac{7}{24})=4\cdot \frac{5}{24}=\frac{5}{6}.$

Between: $\mathrm{Var}(E[X\mid P])=\mathrm{Var}(4P)=16\cdot \frac{1}{24}=\frac{2}{3}.$

By the law of total variance, $\mathrm{Var}(X)=\frac{5}{6}+\frac{2}{3}=\frac{5+4}{6}=\frac{3}{2}$.

A fair coin tossed four times has variance $4\cdot \frac{1}{2}\cdot \frac{1}{2}=1$. The mixture is more variable by $\frac{1}{2}$: not knowing the bias adds a between-group term of $\frac{2}{3}$ that a fixed coin does not have (it also trims the within term slightly, from $1$ to $\frac{5}{6}$, because $p(1-p)$ is largest at $p=\frac{1}{2}$, but the between term dominates).

(b) Variance of a random sum. With $X_i$ the items bought by customer $i$: $\mu =E[X_1]=\frac{1}{3}+\frac{2}{6}=\frac{2}{3}$, $E[X_1^2]=\frac{1}{3}+\frac{4}{6}=1$, so ${\sigma }^2=1-\frac{4}{9}=\frac{5}{9}$. For a Poisson, $E[N]=\mathrm{Var}(N)=6$. By the corollary on the variance of a random sum,

$$\mathrm{Var}(S_N)=E[N]{\sigma }^2+\mathrm{Var}(N){\mu }^2=6\cdot \frac{5}{9}+6\cdot \frac{4}{9}=\frac{30}{9}+\frac{24}{9}=6.$$

The first term, $\frac{10}{3}$, is the within part (uncertainty about how much each customer buys); the second, $\frac{8}{3}$, is the between part (uncertainty about how many customers there are).

In general, when $N$ is Poisson with mean $\lambda$ the two coefficients coincide, $E[N]=\mathrm{Var}(N)=\lambda$, so

$$\mathrm{Var}(S_N)=\lambda {\sigma }^2+\lambda {\mu }^2=\lambda ({\sigma }^2+{\mu }^2)=\lambda E[X_1^2],$$

by the computational formula for the variance. Here $\lambda E[X_1^2]=6\cdot 1=6$. ✓

(c) Direct check via $E[X^2]$. In the conditional world, $E[X^2\mid P]=\mathrm{Var}(X\mid P)+(E[X\mid P]{)}^2=4P(1-P)+16P^2=4P+12P^2$. By the tower property,

$$E[X^2]=4E[P]+12E[P^2]=4\cdot \frac{1}{2}+12\cdot \frac{7}{24}=2+\frac{7}{2}=\frac{11}{2}.$$

Since $E[X]=4E[P]=2$, $\mathrm{Var}(X)=\frac{11}{2}-4=\frac{3}{2}$, agreeing with (a). ✓

(a) The conditional mean and its error. The group means are $m(0)=0$, $m(1)=\frac{1+3}{2}=2$, $m(2)=\frac{0+2}{2}=1$. So $m(Y)$ takes the values $0$, $2$, $1$ with probability $\frac{1}{3}$ each — note it is not monotone in $Y$.

By the theorem that conditional expectation minimises mean squared error, its error is $E[\mathrm{Var}(X\mid Y)]$. The conditional variances are $\mathrm{Var}(X\mid Y=0)=0$, and for the two-point groups $\{1,3\}$ and $\{0,2\}$, each symmetric about its mean at distance $1$, $\mathrm{Var}(X\mid Y=1)=\mathrm{Var}(X\mid Y=2)=1$. Hence

$$E[(X-m(Y){)}^2]=\frac{1}{3}(0+1+1)=\frac{2}{3}.$$

(b) The best line. First the moments. $E[Y]=1$ and $\mathrm{Var}(Y)=\frac{1}{3}(1+0+1)=\frac{2}{3}$. By the tower property $E[X]=E[m(Y)]=\frac{1}{3}(0+2+1)=1$ and, taking out what is known, $E[XY]=E[Ym(Y)]=\frac{1}{3}(0\cdot 0+1\cdot 2+2\cdot 1)=\frac{4}{3}$, so $\mathrm{Cov}(X,Y)=\frac{4}{3}-1=\frac{1}{3}$. For $\mathrm{Var}(X)$, $E[X^2\mid Y]$ takes the values $0$, $\frac{1+9}{2}=5$, $\frac{0+4}{2}=2$, so $E[X^2]=\frac{1}{3}(0+5+2)=\frac{7}{3}$ and $\mathrm{Var}(X)=\frac{7}{3}-1=\frac{4}{3}$.

By the proposition on the best linear predictor,

$$b^{\ast }=\frac{\mathrm{Cov}(X,Y)}{\mathrm{Var}(Y)}=\frac{1/3}{2/3}=\frac{1}{2},a^{\ast }=E[X]-b^{\ast }E[Y]=1-\frac{1}{2}=\frac{1}{2},$$

so the best line is $\frac{1}{2}+\frac{1}{2}Y$, predicting $\frac{1}{2}$, $1$, $\frac{3}{2}$ at $Y=0,1,2$.

Via the formula: ${\rho }^2=\frac{\mathrm{Cov}(X,Y{)}^2}{\mathrm{Var}(X)\mathrm{Var}(Y)}=\frac{1/9}{(4/3)(2/3)}=\frac{1/9}{8/9}=\frac{1}{8}$, so the error is $\mathrm{Var}(X)(1-{\rho }^2)=\frac{4}{3}\cdot \frac{7}{8}=\frac{7}{6}$.

Via the decomposition: the gaps between the line and $m(Y)$ at $Y=0,1,2$ are $\frac{1}{2}-0$, $1-2$, $\frac{3}{2}-1$, with squares $\frac{1}{4}$, $1$, $\frac{1}{4}$ and mean $\frac{1}{3}\cdot \frac{3}{2}=\frac{1}{2}$. Adding the irreducible error from (a), $\frac{2}{3}+\frac{1}{2}=\frac{7}{6}$. ✓ (The cross term vanishes because $X-m(Y)$ is uncorrelated with every function of $Y$, which is what makes the decomposition Pythagorean.)

(c) The best constant. By the corollary on the best constant predictor, the minimiser is $c=E[X]=1$ with error $\mathrm{Var}(X)=\frac{4}{3}$. Ranking the errors: conditional mean $\frac{2}{3}$ $<$ best line $\frac{7}{6}$ $<$ best constant $\frac{4}{3}$. Each enlargement of the class of allowed predictors can only help, and here each one does.

(d) Two different fractions of variance explained. ${\rho }^2=\frac{1}{8}$ is the fraction of $\mathrm{Var}(X)$ removed by the best linear predictor: indeed $\frac{4/3-7/6}{4/3}=\frac{1/6}{4/3}=\frac{1}{8}$.

$\mathrm{Var}(m(Y))=E[m(Y{)}^2]-1=\frac{1}{3}(0+4+1)-1=\frac{2}{3}$, so $\frac{\mathrm{Var}(m(Y))}{\mathrm{Var}(X)}=\frac{2/3}{4/3}=\frac{1}{2}$. By the law of total variance this is the between share — the fraction of $\mathrm{Var}(X)$ removed by the best predictor of any form, since the remaining $\frac{2}{3}$ is exactly the within term.

They differ because $m(Y)$ is not a linear function of $Y$: it goes $0,2,1$, and a straight line can only capture the tilt of that pattern, not its bend. The gap $\frac{1}{2}-\frac{1}{8}=\frac{3}{8}$ of the variance is the price of insisting on a line. Had $m$ been linear in $Y$, the two fractions would coincide.

(a) The numbers. $E[X]=\frac{3}{4}\cdot 1+\frac{1}{4}\cdot 4=\frac{3}{4}+1=\frac{7}{4}$, so $\phi (E[X])=\frac{4}{7}$. And $E[\phi (X)]=E[1/X]=\frac{3}{4}\cdot 1+\frac{1}{4}\cdot \frac{1}{4}=\frac{3}{4}+\frac{1}{16}=\frac{13}{16}$.

Since $\frac{13}{16}=\frac{91}{112}$ and $\frac{4}{7}=\frac{64}{112}$, we have $E[\phi (X)]>\phi (E[X])$, with gap $\frac{27}{112}$. This is Jensen's inequality for a convex $\phi$: on $(0,\infty )$, ${\phi }^{\prime \prime }(x)=2/x^3>0$, so $1/x$ is strictly convex there, and $X$ takes values in $(0,\infty )$ with probability one.

(b) The drawing, feature by feature. The curve $y=1/x$ is decreasing and bends upward (convex) through $(\frac{1}{2},2)$, $(1,1)$, $(2,\frac{1}{2})$, $(4,\frac{1}{4})$, $(5,\frac{1}{5})$.

The chord. It joins $(1,1)$ to $(4,\frac{1}{4})$; its slope is $\frac{1/4-1}{3}=-\frac{1}{4}$ and its equation is $y=1-\frac{1}{4}(x-1)=\frac{5-x}{4}$. Because $\phi$ is convex, the chord lies above the curve on $(1,4)$ — draw it so.

The two marked points above $x=\frac{7}{4}$. On the chord, $y=\frac{5-7/4}{4}=\frac{13}{16}$; on the curve, $y=\frac{4}{7}$. Label them $(\frac{7}{4},\frac{13}{16})$ and $(\frac{7}{4},\frac{4}{7})$; the vertical gap between them is $E[\phi (X)]-\phi (E[X])=\frac{27}{112}$.