Bayesian Inference

(a) Bayes' theorem with the marginal written out. The prior is $P(D)=\frac{1}{10}$, $P(D^c)=\frac{9}{10}$. The likelihoods of a positive result are $P(+\mid D)=\frac{4}{5}$ and $P(+\mid D^c)=1-\frac{9}{10}=\frac{1}{10}$. The marginal probability of a positive is

$$P(+)=P(+\mid D)P(D)+P(+\mid D^c)P(D^c)=\frac{4}{5}\cdot \frac{1}{10}+\frac{1}{10}\cdot \frac{9}{10}=\frac{8}{100}+\frac{9}{100}=\frac{17}{100}.$$

Hence

$$P(D\mid +)=\frac{P(+\mid D)P(D)}{P(+)}=\frac{8/100}{17/100}=\frac{8}{17}.$$

A positive result moves the probability from $\frac{1}{10}$ to a little under one half: of every $17$ positives, $9$ are false, because the healthy nine-tenths of the population generate almost as many false positives as the sick tenth generates true ones.

(b) The odds route. The prior odds of disease are $P(D):P(D^c)=1:9$, that is $\frac{1}{9}$. The likelihood ratio of a positive result is

$$\frac{P(+\mid D)}{P(+\mid D^c)}=\frac{4/5}{1/10}=8,$$

so the posterior odds are $\frac{1}{9}\times 8=\frac{8}{9}$. Converting back, $P(D\mid +)=\frac{8/9}{1+8/9}=\frac{8}{17}$, agreeing with (a). The marginal $P(+)$ never appears on the odds scale — it cancels in the ratio — which is why sequential updating is so convenient there.

(c) A negative result. Now the likelihoods are $P(-\mid D)=1-\frac{4}{5}=\frac{1}{5}$ and $P(-\mid D^c)=\frac{9}{10}$, so the likelihood ratio is $\frac{1/5}{9/10}=\frac{2}{9}$ and the posterior odds are $\frac{1}{9}\cdot \frac{2}{9}=\frac{2}{81}$. Therefore

$$P(D\mid -)=\frac{2/81}{1+2/81}=\frac{2}{83}.$$

Check by the marginal route: $P(-)=\frac{1}{5}\cdot \frac{1}{10}+\frac{9}{10}\cdot \frac{9}{10}=\frac{2}{100}+\frac{81}{100}=\frac{83}{100}$, and $P(D\mid -)=\frac{2/100}{83/100}=\frac{2}{83}$. ✓

A positive multiplies the odds by $8$; a negative multiplies them by $\frac{2}{9}$, that is, divides them by $4\frac{1}{2}$. The positive result is the larger move. The two ratios say why: a positive is $8$ times as likely under disease as under health because the false-positive rate $\frac{1}{10}$ is small, whereas a negative is only $4\frac{1}{2}$ times as likely under health as under disease because the false-negative rate $\frac{1}{5}$ is not so small. A test's two error rates contribute separately, one to each direction of update.

(a) Likelihood times prior. Three heads in three tosses have probability $f(\mathbf{x}\mid \theta )={\theta }^3$, so $\pi (\theta \mid \mathbf{x})\propto {\theta }^3\pi (\theta )$. The unnormalised values are

$$\theta =\frac{1}{4}:\frac{1}{64}\cdot \frac{1}{4}=\frac{1}{256},\theta =\frac{1}{2}:\frac{1}{8}\cdot \frac{1}{2}=\frac{1}{16}=\frac{16}{256},\theta =\frac{3}{4}:\frac{27}{64}\cdot \frac{1}{4}=\frac{27}{256}.$$

The marginal likelihood is the sum, $f(\mathbf{x})={\sum }_{\theta }f(\mathbf{x}\mid \theta )\pi (\theta )=\frac{1+16+27}{256}=\frac{44}{256}=\frac{11}{64}$: the overall probability of seeing three heads, averaged over what the prior believed about the coin.

(b) The posterior. Dividing each unnormalised value by $\frac{44}{256}$,

$$\pi (\frac{1}{4}\mid \mathbf{x})=\frac{1}{44},\pi (\frac{1}{2}\mid \mathbf{x})=\frac{16}{44}=\frac{4}{11},\pi (\frac{3}{4}\mid \mathbf{x})=\frac{27}{44},$$

and $\frac{1+16+27}{44}=1$. ✓

The change factors are posterior over prior: $\frac{1/44}{1/4}=\frac{1}{11}$ for $\theta =\frac{1}{4}$, $\frac{4/11}{1/2}=\frac{8}{11}$ for $\theta =\frac{1}{2}$, and $\frac{27/44}{1/4}=\frac{27}{11}$ for $\theta =\frac{3}{4}$. Two values lost probability and one gained; only $\theta =\frac{3}{4}$ explains three heads better than average. The factor is $f(\mathbf{x}\mid \theta )/f(\mathbf{x})$ because Bayes' theorem reads $\pi (\theta \mid \mathbf{x})=\pi (\theta )\cdot \frac{f(\mathbf{x}\mid \theta )}{f(\mathbf{x})}$: a value is promoted exactly when it predicts the data better than the prior-averaged prediction $f(\mathbf{x})=\frac{11}{64}$. Check: $\frac{1/64}{11/64}=\frac{1}{11}$, $\frac{1/8}{11/64}=\frac{8}{11}$, $\frac{27/64}{11/64}=\frac{27}{11}$. ✓

(c) A prior that excludes a value. The unnormalised values are now $\frac{1}{64}\cdot \frac{1}{2}=\frac{1}{128}$, $\frac{1}{8}\cdot \frac{1}{2}=\frac{8}{128}$ and $\frac{27}{64}\cdot 0=0$, with sum $\frac{9}{128}$, so the posterior is

$$\pi (\frac{1}{4}\mid \mathbf{x})=\frac{1}{9},\pi (\frac{1}{2}\mid \mathbf{x})=\frac{8}{9},\pi (\frac{3}{4}\mid \mathbf{x})=0.$$

After a hundred heads the unnormalised values are ${\left(\frac{1}{4}\right)}^{100}\cdot \frac{1}{2}$, ${\left(\frac{1}{2}\right)}^{100}\cdot \frac{1}{2}$ and $0$, in the ratio $1:2^{100}:0$. So $\pi (\frac{1}{2}\mid \mathbf{x})=\frac{2^{100}}{2^{100}+1}$, essentially $1$, while $\pi (\frac{3}{4}\mid \mathbf{x})$ is still exactly $0$ — even though $\theta =\frac{3}{4}$ is by far the best explanation of the data among the three, and $\theta =\frac{1}{2}$ predicts a hundred heads with probability $2^{-100}$.

The lesson: the posterior is a product, and a factor of zero in the prior survives every multiplication. Data cannot revive a value the prior excluded; the mass is pushed instead onto the best of the values that were allowed. A prior that assigns zero probability to a region is therefore a permanent commitment, and should be made only when the region is genuinely impossible.

(a) Proof. By the definition of the posterior,

$$\pi (\theta \mid x_1,x_2)=\frac{f(x_1,x_2\mid \theta )\pi (\theta )}{f(x_1,x_2)}=\frac{f(x_1\mid \theta )f(x_2\mid \theta )\pi (\theta )}{f(x_1,x_2)},$$

using conditional independence in the numerator. The denominator $f(x_1,x_2)=\int f(x_1,x_2\mid \theta )\pi (\theta )d\theta$ is a number, not a function of $\theta$, so

$$\pi (\theta \mid x_1,x_2)\propto f(x_1\mid \theta )f(x_2\mid \theta )\pi (\theta ).$$

Now apply the definition to the first observation alone: $\pi (\theta \mid x_1)=f(x_1\mid \theta )\pi (\theta )/f(x_1)$, so $f(x_1\mid \theta )\pi (\theta )=f(x_1)\pi (\theta \mid x_1)$, where $f(x_1)$ is again a constant in $\theta$. Substituting,

$$\pi (\theta \mid x_1,x_2)\propto f(x_1)f(x_2\mid \theta )\pi (\theta \mid x_1)\propto f(x_2\mid \theta )\pi (\theta \mid x_1).$$

The two constants dropped are $f(x_1,x_2)$ and $f(x_1)$, both marginal likelihoods and both free of $\theta$. Dropping them loses nothing because a density is determined by its kernel: the left side integrates to one, so its normalising constant is forced to be $1/\int f(x_2\mid \theta )\pi (\theta \mid x_1)d\theta$, and

$$\pi (\theta \mid x_1,x_2)=\frac{f(x_2\mid \theta )\pi (\theta \mid x_1)}{\int f(x_2\mid \theta )\pi (\theta \mid x_1)d\theta },$$

which is exactly Bayes' theorem with $\pi (\theta \mid x_1)$ in the role of the prior. $■$

(b) Order. The kernel $f(x_1\mid \theta )f(x_2\mid \theta )\pi (\theta )$ is unchanged if the labels $1$ and $2$ are swapped, so the argument in (a) with the roles reversed gives $\pi (\theta \mid x_2,x_1)\propto f(x_1\mid \theta )\pi (\theta \mid x_2)$ with the same kernel. Two densities with the same kernel are equal, so the two posteriors coincide. Yesterday's posterior is today's prior, and it does not matter which day came first.

(c) Beta-binomial. By Beta-binomial conjugacy, after the first experiment $\theta \mid x_1\sim \text{Beta}(a+x_1,b+n_1-x_1)$. Using this as the prior for the second experiment and applying the same theorem,

$$\theta \mid x_1,x_2\sim \text{Beta}(a+x_1+x_2,b+n_1-x_1+n_2-x_2)=\text{Beta}(a+(x_1+x_2),b+(n_1+n_2)-(x_1+x_2)),$$

which is precisely the posterior from a single experiment with $x_1+x_2$ successes in $n_1+n_2$ trials. At the level of kernels the same thing is visible at once:

$${\theta }^{x_2}(1-\theta {)}^{n_2-x_2}\cdot {\theta }^{a+x_1-1}(1-\theta {)}^{b+n_1-x_1-1}={\theta }^{a+x_1+x_2-1}(1-\theta {)}^{b+n_1+n_2-x_1-x_2-1}.$$

The updating rule “add successes to $a$, failures to $b$” is additive, and addition does not care about order.

(a) Update by conjugacy. By the theorem that the Beta family is conjugate to the binomial, add the $12$ successes to $a$ and the $8$ failures to $b$:

$$\theta \mid x\sim \text{Beta}(3+12,3+8)=\text{Beta}(15,11),E(\theta \mid x)=\frac{15}{15+11}=\frac{15}{26}.$$

(b) Weighted-average form. With $a+b=6$ and $n=20$, the weight on the data is

$$w=\frac{n}{n+a+b}=\frac{20}{26}=\frac{10}{13},$$

and $\hat{p}=\frac{12}{20}=\frac{3}{5}$, ${\mu }_0=\frac{3}{6}=\frac{1}{2}$. Then

$$w\hat{p}+(1-w){\mu }_0=\frac{10}{13}\cdot \frac{3}{5}+\frac{3}{13}\cdot \frac{1}{2}=\frac{6}{13}+\frac{3}{26}=\frac{12}{26}+\frac{3}{26}=\frac{15}{26}.✓$$

The posterior mean sits between $\frac{1}{2}$ and $\frac{3}{5}$, ten-thirteenths of the way toward the data.

(c) When the prior stops mattering. The prior's weight is $1-w=\frac{a+b}{n+a+b}=\frac{6}{n+6}$, and

$$\frac{6}{n+6}<\frac{1}{50}⟺300<n+6⟺n>294,$$

so the smallest such $n$ is $295$. (At that point the posterior mean is $\frac{3+\frac{3}{5}\cdot 295}{301}=\frac{180}{301}$, against a sample proportion of $\frac{180.6}{301}$ — the prior is nudging the third decimal.) The weight is computable before any data arrive, which is what makes the prior an assumption one can report rather than hide.

(d) Prior strength in units of data. A $\text{Beta}(a,b)$ prior behaves like $a$ previously seen successes and $b$ failures, so $\text{Beta}(3,3)$ is worth $a+b=6$ observations and $\text{Beta}(30,30)$ is worth $60$. With the stronger prior the posterior is $\text{Beta}(42,38)$ with mean

$$\frac{42}{80}=\frac{21}{40},w=\frac{20}{80}=\frac{1}{4}.$$

The same $20$ items move the weak prior's mean from $\frac{1}{2}$ to $\frac{15}{26}\approx 0.58$ but the strong prior's only to $\frac{21}{40}=0.525$: the data are outweighed three to one by sixty pseudo-observations. Both are legitimate posteriors; what differs is how much prior information the analyst has claimed, and a report should say which it was.

(a) Posterior kernel. Three successes in three trials have likelihood $L(\theta )={\theta }^3$. Multiplying by the prior,

$$\pi (\theta \mid \mathbf{x})\propto {\theta }^3\cdot \theta (1-\theta )={\theta }^4(1-\theta ),$$

which is the kernel ${\theta }^{5-1}(1-\theta {)}^{2-1}$ of a $\text{Beta}(5,2)$ — in agreement with the updating rule $\text{Beta}(2+3,2+0)$. Integrating the kernel,

$${\int }_0^1{\theta }^4(1-\theta )d\theta =\frac{1}{5}-\frac{1}{6}=\frac{1}{30},$$

so the posterior density is $\pi (\theta \mid \mathbf{x})=30{\theta }^4(1-\theta )$.

(b) Three summaries. The likelihood ${\theta }^3$ is increasing on $[0,1]$, so the MLE is $\hat{\theta }=1$. For the mode, differentiate the kernel: $\frac{d}{d\theta }{\theta }^4(1-\theta )=4{\theta }^3-5{\theta }^4={\theta }^3(4-5\theta )$, which is positive for $\theta <\frac{4}{5}$ and negative beyond, so the mode is $\frac{4}{5}$ (the Beta mode formula $\frac{\alpha -1}{\alpha +\beta -2}=\frac{4}{5}$ agrees). The mean of a $\text{Beta}(5,2)$ is $\frac{5}{7}$. The order is

$$\text{mean }\frac{5}{7}<\text{mode }\frac{4}{5}<\text{MLE }1,$$

the mean pulled furthest toward the prior's centre $\frac{1}{2}$ because it averages over the whole posterior, including its long left shoulder, while the mode tracks the likelihood most closely.

(c) The sketch. Three curves on $[0,1]$.

Prior $6\theta (1-\theta )$: zero at both ends, a symmetric hump peaking at $\theta =\frac{1}{2}$ with height $6\cdot \frac{1}{4}=\frac{3}{2}$.

Rescaled likelihood $4{\theta }^3$ (since ${\int }_0^1{\theta }^{3}d\theta =\frac{1}{4}$): zero at $\theta =0$, increasing and convex throughout, reaching its maximum $4$ at the right-hand endpoint $\theta =1$. Its “peak” is the endpoint — the data alone say the coin might as well always succeed.

Posterior $30{\theta }^4(1-\theta )$: zero at $\theta =0$, where it leaves the axis very flat (order ${\theta }^4$), rises to its peak at $\theta =\frac{4}{5}$ of height $30\cdot \frac{256}{625}\cdot \frac{1}{5}=\frac{1536}{625}\approx 2.46$, then drops steeply to zero at $\theta =1$. For reference it passes through height $30\cdot \frac{1}{16}\cdot \frac{1}{2}=\frac{15}{16}$ at $\theta =\frac{1}{2}$, already below the prior there, and it is visibly narrower than the prior — combining two sources of information sharpens the result.

The posterior's peak lies to the left of the likelihood's, at $\frac{4}{5}$ rather than $1$, because the prior factor $(1-\theta )$ penalises values near $1$ and the compromise lands between the prior's centre and the data's endpoint. It lies far closer to the data end because three certain successes are strong evidence against small $\theta$, whereas the prior, worth only four observations, is weak.

The posterior is exactly zero at $\theta =1$ because the prior is: $\pi (1)=6\cdot 1\cdot 0=0$, and the posterior is the product likelihood $\times$ prior, so a zero in either factor is a zero in the product. The likelihood being maximal there cannot help — $1\times 0=0$. This is the prior's veto at work; it is tolerable here only because a prior density that vanishes at a single endpoint still gives positive probability to every neighbourhood of it, so the posterior can pile up arbitrarily close to $1$ as successes accumulate.

(a) Precisions add. By normal-normal conjugacy the posterior is normal with precision equal to the sum of the prior and data precisions:

$$\frac{1}{{\tau }_n^2}=\frac{1}{{\tau }^2}+\frac{n}{{\sigma }^2}=\frac{1}{25}+\frac{21}{100}=\frac{4}{100}+\frac{21}{100}=\frac{25}{100}=\frac{1}{4},$$

so the posterior variance is ${\tau }_n^2=4$ and the posterior standard deviation is $2$. The posterior mean is

$${\mu }_n={\tau }_n^2\left(\frac{{\mu }_0}{{\tau }^2}+\frac{n\bar{x}}{{\sigma }^2}\right)=4\left(\frac{50}{25}+\frac{21\cdot 60}{100}\right)=4\left(2+\frac{63}{5}\right)=4\cdot \frac{73}{5}=\frac{292}{5}=\mathrm{58.4.}$$

So $\theta \mid \mathbf{x}\sim N(58.4,4)$.

(b) Precision-weighted average. Each source is weighted by its share of the total precision: the data get $\frac{21/100}{25/100}=\frac{21}{25}$ and the prior gets $\frac{4/100}{25/100}=\frac{4}{25}$. Then

$$\frac{21}{25}\cdot 60+\frac{4}{25}\cdot 50=\frac{1260}{25}+\frac{200}{25}=\frac{1460}{25}=\mathrm{58.4.}✓$$

The answer is pulled $\frac{4}{25}$ of the way from the sample mean back toward the prior mean, because the twenty-one measurements carry more than five times the prior's precision.

(c) Credible against confidence. The $95\%$ equal-tailed credible interval is

$$58.4\pm 1.96\times 2=58.4\pm 3.92=(54.48,62.32).$$

It licenses the direct statement: given the prior and these data, $P(54.48<\theta <62.32\mid \mathbf{x})=0.95$.

The confidence interval treats $\theta$ as fixed and uses only the data: it is centred at $\bar{x}=60$ with half-width $1.96\sigma /\sqrt{n}=1.96\times \frac{10}{\sqrt{21}}$. Since $\frac{100}{21}>4$, the standard error $\frac{10}{\sqrt{21}}$ exceeds the posterior standard deviation $2$, so the confidence interval is the wider of the two — the posterior has borrowed precision from the prior, and its standard deviation is smaller than both the prior's $5$ and the data's $\frac{10}{\sqrt{21}}$, as precision addition guarantees. What the confidence interval does not license is the sentence above: it either contains $\theta$ or it does not, and the $95\%$ describes the coverage of the procedure across repeated samples, not this interval.

(d) Enough data, and no prior at all. The posterior standard deviation is at most $1$ exactly when the posterior precision is at least $1$:

$$\frac{1}{25}+\frac{n}{100}\ge 1⟺\frac{n}{100}\ge \frac{24}{25}⟺n\ge 96.$$

So $96$ measurements in total, that is $75$ more.

As ${\tau }^2\to \infty$ the prior precision $\frac{1}{{\tau }^2}\to 0$, so the posterior precision tends to $\frac{n}{{\sigma }^2}$ and the mean to $\bar{x}$: the posterior becomes $N\left(60,\frac{100}{21}\right)$, and the credible interval becomes $60\pm 1.96\frac{10}{\sqrt{21}}$ — numerically identical to the classical $z$-interval for a normal mean with known variance. This is the case in which the two frameworks agree exactly in their arithmetic and differ only in what the interval is said to mean.

(a) Posterior and its summaries. By Beta-binomial conjugacy the posterior is $\text{Beta}(1+4,1+2)=\text{Beta}(5,3)$. Its mean is $\frac{5}{8}$ and its mode is $\frac{\alpha -1}{\alpha +\beta -2}=\frac{4}{6}=\frac{2}{3}$. The MLE is the sample proportion $\frac{4}{6}=\frac{2}{3}$.

The mode and the MLE coincide, by the proposition that the posterior mode is the MLE under a flat prior: with $\pi (\theta )$ constant the posterior is proportional to the likelihood, and proportional functions peak at the same place. The mean does not coincide — the flat prior is worth two pseudo-observations, one success and one failure, and the mean is pulled from $\frac{2}{3}$ toward $\frac{1}{2}$, to $\frac{5}{8}$. “Flat” is not “no influence”.

(b) The next trial. By the theorem on predicting the next Bernoulli trial, $P(\tilde{X}=1\mid \mathbf{x})=E(\theta \mid \mathbf{x})=\frac{5}{8}$.

(c) Two future trials. The plug-in answer fixes $\theta =\frac{5}{8}$ and multiplies: ${\left(\frac{5}{8}\right)}^2=\frac{25}{64}$.

Correctly, the two trials share the same unknown $\theta$, so use sequential updating. The first succeeds with probability $\frac{5}{8}$ by (b). Conditional on that success the posterior becomes $\text{Beta}(6,3)$, whose mean is $\frac{6}{9}=\frac{2}{3}$, so the second then succeeds with probability $\frac{2}{3}$. Hence

$$P(\text{both})=\frac{5}{8}\cdot \frac{2}{3}=\frac{10}{24}=\frac{5}{12}.$$

Comparing over the common denominator $192$: $\frac{5}{12}=\frac{80}{192}$ and $\frac{25}{64}=\frac{75}{192}$, so the predictive answer is larger. The reason is that a first success is evidence that $\theta$ is high, which raises the chance of the second; the two future trials are positively correlated once $\theta$ is unknown, and plug-in destroys that by pretending $\theta$ is known. Formally $P(\text{both})=E({\theta }^2\mid \mathbf{x})$ while plug-in gives $E(\theta \mid \mathbf{x}{)}^2$, and the difference is $\text{Var}(\theta \mid \mathbf{x})$. Check: $\frac{80-75}{192}=\frac{5}{192}$, and the variance of a $\text{Beta}(5,3)$ is $\frac{\alpha \beta }{(\alpha +\beta {)}^2(\alpha +\beta +1)}=\frac{15}{64\cdot 9}=\frac{15}{576}=\frac{5}{192}$. ✓

(d) Three future trials. Plug-in: $\text{Binomial}(3,\frac{5}{8})$ has variance $3\cdot \frac{5}{8}\cdot \frac{3}{8}=\frac{45}{64}$. Posterior predictive, with $\alpha +\beta =8$ and $m=3$:

$$\frac{45}{64}\cdot \frac{8+3}{8+1}=\frac{45}{64}\cdot \frac{11}{9}=\frac{55}{64}.$$

The predictive variance is larger by the factor $\frac{11}{9}$, about $22\%$: it carries the uncertainty about $\theta$ on top of the trial-to-trial noise. The factor tends to $1$ as $\alpha +\beta$ grows, that is, as the posterior sharpens; with six observations it is far from $1$.

(a) The flaws.

Step 1. The posterior $\text{Beta}(3,3)$ is correct by Beta-binomial conjugacy. The rest is false. A $\text{Beta}(1,1)$ prior is not without influence: it is worth $a+b=2$ observations, and with $n=4$ the data's weight in the posterior mean is only $w=\frac{4}{4+2}=\frac{2}{3}$ — a third of the answer is prior. Nor is flatness a neutral choice: a prior flat in $\theta$ is not flat in $\log \theta$ or in the odds $\frac{\theta }{1-\theta }$, so “know nothing on this scale” is a definite opinion on every other. The result being overreached is the definition of an uninformative prior — “intended to let the data speak” is an intention, not a property — and the conclusion that nothing need be reported is exactly backwards for $n=4$.

Step 2. The mode computation is correct, and mode $=$ MLE is exactly what the proposition on the posterior mode under a flat prior guarantees. But that proposition is about one point summary, and the step stretches it to “the Bayesian and frequentist answers coincide”, which it does not say and which is false in general (the mean and the intervals are different objects). The second sentence then commits the central misreading of the whole subject: a confidence interval is a statement about a procedure's long-run coverage, and once computed it either contains $\theta$ or does not. Only a credible interval — the region between the posterior's $0.025$ and $0.975$ quantiles — licenses “$95\%$ probability that $\theta$ lies in it”. The two intervals are numerically close for a flat prior, which is why the error so rarely shows, but closeness of numbers is not identity of meaning, and the student has not even computed the credible interval.

Step 3. The first claim is correct: by the theorem on predicting the next Bernoulli trial, $P(\tilde{X}=1\mid \mathbf{x})=E(\theta \mid \mathbf{x})=\frac{1}{2}$. The second is the plug-in error: fixing $\theta$ at $\frac{1}{2}$ treats it as known. The number of successes in the next $4$ trials has the posterior predictive distribution, which is beta-binomial and over-dispersed relative to any binomial, because all four future trials share the same uncertain $\theta$ and are therefore positively correlated.

(b) The repair.

Prior. Report it. The posterior mean $\frac{1}{2}$ is $\frac{2}{3}\cdot \frac{1}{2}+\frac{1}{3}\cdot \frac{1}{2}$ — here the prior and the data happen to agree, so the influence is invisible in the mean, but it is visible in the spread and would be visible in the mean for any other data. A sensitivity analysis under, say, $\text{Beta}(\frac{1}{2},\frac{1}{2})$ (the Jeffreys prior, posterior $\text{Beta}(\frac{5}{2},\frac{5}{2})$) would show how little or much the choice matters.

Interval. Either compute the credible interval from the $\text{Beta}(3,3)$ quantiles and make the probability statement, or keep the confidence interval and make the procedural one: “an interval built this way contains $\theta$ in $95\%$ of repeated samples”. Not both from one interval.

Next four trials. The success probability of the next single trial is $\frac{1}{2}$. The count $Y$ over the next $m=4$ trials has mean $4\cdot \frac{1}{2}=2$ but variance

$$m\bar{\theta }(1-\bar{\theta })\cdot \frac{\alpha +\beta +m}{\alpha +\beta +1}=4\cdot \frac{1}{2}\cdot \frac{1}{2}\cdot \frac{6+4}{6+1}=\frac{10}{7},$$

not $1$: some $43\%$ more spread than the binomial the student wrote down. The inflation is large precisely because the posterior is wide — six pseudo-observations in all — which is when the plug-in error costs the most.

(a) Four heads. Under $M_0$ the four heads have probability $P(\mathbf{x}\mid M_0)={\left(\frac{1}{2}\right)}^4=\frac{1}{16}$. Under $M_1$ the marginal likelihood averages ${\theta }^4$ over the uniform prior:

$$P(\mathbf{x}\mid M_1)={\int }_0^1{\theta }^{4}d\theta =\frac{4!0!}{5!}=\frac{1}{5}.$$

Hence

$${\text{BF}}_{10}=\frac{1/5}{1/16}=\frac{16}{5}=3\frac{1}{5}.$$

The data are $3\frac{1}{5}$ times as probable under the biased-coin model as under the fair coin: whatever odds a reader held between the two models beforehand, four straight heads multiply them by $\frac{16}{5}$ in favour of $M_1$. That is moderate, not decisive, evidence.

(b) All heads in $n$ tosses. Under $M_0$ the probability is $2^{-n}$; under $M_1$ it is ${\int }_0^1{\theta }^{n}d\theta =\frac{n!0!}{(n+1)!}=\frac{1}{n+1}$. So

$${\text{BF}}_{10}=\frac{1/(n+1)}{2^{-n}}=\frac{2^n}{n+1},$$

which for $n=8$ is $\frac{256}{9}\approx 28$. The fair coin's prediction shrinks geometrically while the uniform alternative's shrinks only like $\frac{1}{n}$, because $M_1$ keeps a fixed share of its prior mass near $\theta =1$, where an unbroken run is not surprising.

(c) Combining with prior odds. Posterior odds are ${\text{BF}}_{10}\times$ prior odds, with prior odds $\frac{P(M_1)}{P(M_0)}=\frac{1}{4}$.

After four heads: posterior odds $\frac{16}{5}\cdot \frac{1}{4}=\frac{4}{5}$, so $P(M_1\mid \mathbf{x})=\frac{4/5}{1+4/5}=\frac{4}{9}$. The sceptic still leans, just, toward fairness.

After eight heads: posterior odds $\frac{256}{9}\cdot \frac{1}{4}=\frac{64}{9}$, so $P(M_1\mid \mathbf{x})=\frac{64/9}{1+64/9}=\frac{64}{73}$. The prior of $P(M_1)=\frac{1}{5}$ has been moved to almost $0.88$. Note the division of labour: the Bayes factor is the same for every reader, and each reader supplies their own prior odds.

(d) Three heads in four. Under $M_0$: $\left(\genfrac{}{}{0ex}{}{4}{3}\right){\left(\frac{1}{2}\right)}^4=\frac{4}{16}=\frac{1}{4}$. Under $M_1$:

$$P(\mathbf{x}\mid M_1)=\left(\genfrac{}{}{0ex}{}{4}{3}\right){\int }_0^1{\theta }^3(1-\theta )d\theta =4\cdot \frac{3!1!}{5!}=4\cdot \frac{6}{120}=\frac{1}{5}.$$

So ${\text{BF}}_{10}=\frac{1/5}{1/4}=\frac{4}{5}$: the data favour the fair coin, five to four, even though three heads in four is above one half.

The reason is where $M_1$ has committed its prior mass. A uniform prior spends as much of its prediction on $\theta$ near $0$ or $0.1$ — values under which three heads in four are nearly impossible — as on values near $\frac{3}{4}$ that fit well. Its marginal likelihood is an average over all of them, and the badly fitting values drag it down to $\frac{1}{5}$, below the point null's $\frac{1}{4}$. That is the automatic penalty for a diffuse model: it must fit substantially better than the simple one to overcome the cost of having hedged so widely. Four heads in four escaped the penalty only because the data were extreme enough that $M_0$'s own prediction had collapsed to $\frac{1}{16}$.

(a) Every count is equally likely under the uniform alternative. The marginal likelihood of the count $x$ under $M_1$ averages the binomial probability over the prior:

$$P(X=x\mid M_1)={\int }_0^1\left(\genfrac{}{}{0ex}{}{n}{x}\right){\theta }^x(1-\theta {)}^{n-x}d\theta =(\genfrac{}{}{0ex}{}{n}{x})\cdot \frac{x!(n-x)!}{(n+1)!}=\frac{n!}{x!(n-x)!}\cdot \frac{x!(n-x)!}{(n+1)!}=\frac{n!}{(n+1)!}=\frac{1}{n+1}.$$

The factorials from the binomial coefficient cancel exactly against those from the Beta integral, and what remains does not depend on $x$. $■$

Interpretation: before seeing the coin, a uniform prior on $\theta$ makes every possible number of heads from $0$ to $n$ equally probable — the prior predictive distribution of the count is uniform on $\{0,1,\dots ,n\}$. This is what “spreading the prior thinly” means concretely: $M_1$ has hedged across all $n+1$ outcomes equally, so it can never predict any single outcome with probability better than $\frac{1}{n+1}$.

(b) The Bayes factor. Under $M_0$, $P(X=x\mid M_0)=\left(\genfrac{}{}{0ex}{}{n}{x}\right)2^{-n}$. Dividing,

$${\text{BF}}_{10}=\frac{P(X=x\mid M_1)}{P(X=x\mid M_0)}=\frac{1/(n+1)}{\left(\genfrac{}{}{0ex}{}{n}{x}\right)2^{-n}}=\frac{2^n}{(n+1)\left(\genfrac{}{}{0ex}{}{n}{x}\right)}.$$

(As checks, $x=n$ gives $\frac{2^n}{n+1}$, and $n=4$, $x=3$ gives $\frac{16}{5\cdot 4}=\frac{4}{5}$.)

(c) Balanced data. With $x=\frac{n}{2}$,

$$b_n=\frac{2^n}{(n+1)\left(\genfrac{}{}{0ex}{}{n}{n/2}\right)}.$$

The first four values: $b_2=\frac{4}{3\cdot 2}=\frac{2}{3}$; $b_4=\frac{16}{5\cdot 6}=\frac{8}{15}$; $b_6=\frac{64}{7\cdot 20}=\frac{16}{35}$; $b_8=\frac{256}{9\cdot 70}=\frac{128}{315}$.

The ratio. Write $k=\frac{n}{2}$. The central binomial coefficients are related by

$$\left(\genfrac{}{}{0ex}{}{n+2}{k+1}\right)=\frac{(n+2)!}{(k+1)!(k+1)!}=\frac{(n+2)(n+1)}{(k+1{)}^2}\cdot \frac{n!}{k!k!}=\frac{(n+2)(n+1)}{{\left(\frac{n+2}{2}\right)}^2}\left(\genfrac{}{}{0ex}{}{n}{k}\right)=\frac{4(n+1)}{n+2}\left(\genfrac{}{}{0ex}{}{n}{k}\right).$$

Therefore

$$\frac{b_{n+2}}{b_n}=\frac{2^{n+2}}{(n+3)\left(\genfrac{}{}{0ex}{}{n+2}{k+1}\right)}\cdot \frac{(n+1)\left(\genfrac{}{}{0ex}{}{n}{k}\right)}{2^n}=\frac{4(n+1)}{(n+3)}\cdot \frac{n+2}{4(n+1)}=\frac{n+2}{n+3}.■$$

Check against the values: $\frac{b_4}{b_2}=\frac{8/15}{2/3}=\frac{4}{5}=\frac{2+2}{2+3}$ ✓, $\frac{b_6}{b_4}=\frac{16/35}{8/15}=\frac{6}{7}$ ✓, $\frac{b_8}{b_6}=\frac{128/315}{16/35}=\frac{8}{9}$ ✓.

Since $\frac{n+2}{n+3}<1$ for every $n$, each $b_{n+2}<b_n$: the sequence is strictly decreasing, and telescoping the ratios gives $b_n=\frac{2}{3}\cdot \frac{4}{5}\cdot \frac{6}{7}\cdots \frac{n}{n+1}$, a product of factors each below $1$.

Interpretation. ${\text{BF}}_{10}<1$ throughout, so perfectly balanced data always favour the fair coin, and the support for $M_0$ strengthens as the balanced sample grows — a Bayes factor, unlike a $p$-value, can accumulate evidence for a null. But it accumulates slowly: two more balanced tosses multiply the odds for fairness by only $\frac{n+3}{n+2}$, a factor that tends to $1$. The reason is that $M_1$ also predicts balanced data tolerably well, since its prior keeps mass near $\theta =\frac{1}{2}$; the point null wins only by the margin between “exactly $\frac{1}{2}$” and “somewhere near $\frac{1}{2}$”, and that margin, measured by the ratio of $M_0$'s sharp prediction to $M_1$'s flat $\frac{1}{n+1}$, grows like $\sqrt{n}$ rather than geometrically. Contrast an unbroken run of heads, where the Bayes factor $\frac{2^n}{n+1}$ grows exponentially against the null: evidence against a point null comes fast, evidence for it comes slowly.

(a) Acceptance probabilities and the matrix. By the Metropolis rule, a proposal ${\theta }^{\prime }$ is accepted with probability $\alpha =\min \{1,g({\theta }^{\prime })/g(\theta )\}$ — the ratio of posteriors, in which any constant cancels, so $g$ may stand in for $\pi$.

From $1$: propose $2$ with probability $\frac{1}{2}$, accept with $\min \{1,\frac{2}{1}\}=1$, so $P(1,2)=\frac{1}{2}$; propose $0$ with probability $\frac{1}{2}$, rejected. Hence $P(1,1)=\frac{1}{2}$.

From $2$: propose $1$, accept with $\min \{1,\frac{1}{2}\}=\frac{1}{2}$, so $P(2,1)=\frac{1}{2}\cdot \frac{1}{2}=\frac{1}{4}$; propose $3$, accept with $\min \{1,\frac{3}{2}\}=1$, so $P(2,3)=\frac{1}{2}$. Hence $P(2,2)=1-\frac{1}{4}-\frac{1}{2}=\frac{1}{4}$.

From $3$: propose $2$, accept with $\min \{1,\frac{2}{3}\}=\frac{2}{3}$, so $P(3,2)=\frac{1}{2}\cdot \frac{2}{3}=\frac{1}{3}$; propose $4$, rejected. Hence $P(3,3)=\frac{2}{3}$.

$$P=\left(\begin{array}{ccc}\frac{1}{2}& \frac{1}{2}& 0\\ \frac{1}{4}& \frac{1}{4}& \frac{1}{2}\\ 0& \frac{1}{3}& \frac{2}{3}\end{array}\right).$$

Each row sums to one, and the diagonal entries are where rejected proposals went: the chain repeats its current value, and that repeat counts as a draw.

(b) Detailed balance with the unnormalised posterior. For the pair $\{1,2\}$: $g(1)P(1,2)=1\cdot \frac{1}{2}=\frac{1}{2}$ and $g(2)P(2,1)=2\cdot \frac{1}{4}=\frac{1}{2}$. ✓ For $\{2,3\}$: $g(2)P(2,3)=2\cdot \frac{1}{2}=1$ and $g(3)P(3,2)=3\cdot \frac{1}{3}=1$. ✓ For $\{1,3\}$: $P(1,3)=P(3,1)=0$, so both sides are $0$. ✓

Detailed balance holds for $g$, and multiplying every $g$ by the same constant $c$ multiplies both sides of each equation by $c$, so it holds for the normalised $\pi =g/\sum g$ as well. The constant was never needed because the algorithm only ever asks for ratios $g({\theta }^{\prime })/g(\theta )$, from which ${\sum }_{\theta }g(\theta )$ cancels. Here the constant is a sum of three numbers; for a real model it is an integral over a high-dimensional parameter space with no closed form, and being able to sample without ever evaluating it is exactly what made Bayesian computation practical.

(c) Stationarity directly. The normalised posterior is $\pi =\left(\frac{1}{6},\frac{2}{6},\frac{3}{6}\right)=\left(\frac{1}{6},\frac{1}{3},\frac{1}{2}\right)$. Multiplying,

$$(\pi P{)}_1=\frac{1}{6}\cdot \frac{1}{2}+\frac{1}{3}\cdot \frac{1}{4}+\frac{1}{2}\cdot 0=\frac{1}{12}+\frac{1}{12}=\frac{1}{6},$$

$$(\pi P{)}_2=\frac{1}{6}\cdot \frac{1}{2}+\frac{1}{3}\cdot \frac{1}{4}+\frac{1}{2}\cdot \frac{1}{3}=\frac{1}{12}+\frac{1}{12}+\frac{1}{6}=\frac{1}{3},$$

$$(\pi P{)}_3=\frac{1}{6}\cdot 0+\frac{1}{3}\cdot \frac{1}{2}+\frac{1}{2}\cdot \frac{2}{3}=\frac{1}{6}+\frac{1}{3}=\frac{1}{2}.$$

So $\pi P=\pi$: the posterior is stationary, as the Metropolis theorem promises via detailed balance. Because the chain can reach every state from every other and can stay put, it is also irreducible and aperiodic, so long-run draws are distributed as $\pi$ whatever the start.

(d) Monte Carlo error. The exact posterior mean is

$$E(\theta \mid \mathbf{x})=\frac{1\cdot 1+2\cdot 2+3\cdot 3}{6}=\frac{14}{6}=\frac{7}{3},$$

and $E({\theta }^2\mid \mathbf{x})=\frac{1+8+27}{6}=6$, so the posterior variance is $6-\frac{49}{9}=\frac{5}{9}$ and the standard deviation is $\frac{\sqrt{5}}{3}$.

The Monte Carlo standard error of the estimated mean uses the effective sample size, not the number of iterations:

$$\text{MCSE}=\frac{\sqrt{5}/3}{\sqrt{400}}=\frac{\sqrt{5}}{60}.$$

Treating all $10,000$ correlated draws as independent would give $\frac{\sqrt{5}/3}{100}=\frac{\sqrt{5}}{300}$, five times too small — the factor is $\sqrt{10,000/400}=5$. Even the correct figure is only $\frac{1}{20}$ of the posterior standard deviation, so the simulation is adding little to the uncertainty the data leave; that comparison, not the raw iteration count, is what decides whether the chain has run long enough.

(a) The flaws, in the order they occur.

One chain, one start, no burn-in. Starting at the MLE is a sensible choice of starting point, but it does not establish that the chain has forgotten it, and “no burn-in was needed” is asserted rather than checked. More seriously, a single chain cannot show convergence at all: a chain stuck in one region of the posterior looks perfectly stable from the inside. The ignored check is to run several chains from dispersed starting points, confirm that they mix into the same region, and compute the Gelman–Rubin statistic $\hat{R}$, which compares between-chain and within-chain variance and should be near $1$. What is confused is stability of one chain with convergence to the posterior.

Tiny steps and a $98\%$ acceptance rate. A high acceptance rate is not evidence of a healthy sampler. With a very small proposal step every proposed ${\theta }^{\prime }$ is close to $\theta$, so the posterior ratio is close to $1$ and nearly everything is accepted — but each move is tiny, consecutive draws are nearly identical, and the chain crawls across the posterior. That is a chain with very high autocorrelation and hence a small effective sample size; the report reads a symptom of the problem as a sign of success. What is confused is accepting proposals with exploring the posterior.

The Monte Carlo standard error. The formula $s/\sqrt{N}$ requires $N$ independent draws. Draws from a Markov chain are not independent, and the effective number of independent draws is smaller than the number of iterations by a factor depending on the autocorrelation — here, by the report's own admission of tiny steps, a large factor. The correct denominator is $\sqrt{\text{ESS}}$. What is confused is iterations with effective sample size.

The credible interval. This is the gravest error. A credible interval describes where $\theta$ lies, so its width is set by the posterior's own spread, the standard deviation $3$. The Monte Carlo standard error describes how accurately the simulation has located the posterior mean, and it shrinks to zero as the chain runs longer, whereas the posterior's spread does not shrink at all — it reflects genuine uncertainty that no amount of computing removes. Reporting mean $\pm 0.06$ claims to know $\theta$ to within a tenth of a unit when the data themselves leave it uncertain to within several units. What is confused is uncertainty about $\theta$ with uncertainty about a simulated summary of it — the same distinction as between a posterior and its Monte Carlo estimate.

(b) The repair.

Procedure. Run at least three or four chains from widely dispersed starting values, discard the initial stretch of each as burn-in, and check that $\hat{R}\approx 1$ and that the trace plots overlap. Increase the proposal step until the chain moves in appreciable jumps — an acceptance rate well below $98\%$ is the price of a chain that actually travels — and estimate the effective sample size from the autocorrelation of the pooled draws. Then report the posterior summaries with their Monte Carlo errors alongside.

Numbers. With effective sample size $100$ the Monte Carlo standard error of the posterior mean is

$$\frac{3}{\sqrt{100}}=\frac{3}{10}=0.3,$$

ten times the reported $0.03$. (The posterior standard deviation $3$ is itself estimated from correlated draws; it remains a consistent estimate, but its own Monte Carlo error is likewise inflated.)