1.1 What is a random variable?

Before we can speak of Bayesian inference we need to be clear about what a probability is meant to describe. There are two common answers, and the difference between them is the root of most of what distinguishes Bayesian statistics from the methods you may have met before.

In the frequentist view, a random variable is the outcome of an experiment or game that could, at least in principle, be repeated indefinitely under the same conditions. The probability of an outcome is the long-run proportion of repetitions in which it occurs. This is how Fermat and Pascal thought about games of dice in the seventeenth century, and it remains a natural way to think about coins, cards and many physical processes. On this view a probability is a property of the system being studied, in the same way that its mass or length is.

In the Bayesian view, a random variable is any quantity whose value is unknown to us, and its probability distribution describes our uncertainty about it. A probability is then a property of a person, or more precisely of a person’s state of information, rather than of the system alone. Two people with different information may quite reasonably assign different probabilities to the same event.

The Bayesian view is the wider of the two. The number of heads in ten tosses of a coin is a random variable on either view. But consider the proportion of voters in Scotland who favoured independence in September 2014, the speed of light, or whether a particular patient has a particular disease. Each of these has a definite value. Nothing about them is repeatable, and on the frequentist view it makes no sense to give them probability distributions. We are nonetheless uncertain about them, and the Bayesian is content to describe that uncertainty by a probability distribution. This is what allows a Bayesian to make statements such as ‘the probability that the disease is present is 0.05’ or ‘there is a 95% probability that the true value lies in this interval’, which have no direct frequentist meaning.

1.2 Learning with Bayes’ theorem

Bayesian inference is the process of learning from data by means of Bayes’ theorem. Schematically, $$\text{current beliefs} + \text{new information} \xrightarrow{\ \text{Bayes' theorem}\ } \text{updated beliefs}.$$ In most statistical problems the beliefs in question concern the values of parameters that describe a population or a process: the mean height of people in York, say, or the rate at which a machine fails. The new information usually takes the form of a sample of observations whose distribution depends on those parameters. Beliefs, both before and after the data are seen, are expressed as probability distributions over the possible values of the parameters.

We shall use the following notation throughout.

• $\theta$ denotes the parameter of interest, and $\Theta$ the set of its possible values, called the parameter space. The parameter may be a single number or a vector.

• $y$ denotes the observed data, and $\mathcal{Y}$ the set of possible observations, called the sample space.

• $p(\theta)$ is the prior distribution. It describes our beliefs about $\theta$ before the data are seen.

• $p(y\mid\theta)$ is the sampling model. For each possible value of $\theta$ it gives the probability (or probability density) of observing the data $y$ if $\theta$ were the true value. When we fix $y$ at the value actually observed and regard $p(y\mid\theta)$ as a function of $\theta$, we call it the likelihood.

• $p(\theta\mid y)$ is the posterior distribution. It describes our beliefs about $\theta$ after the data have been seen.

The prior and the sampling model together specify a joint distribution for $\theta$ and $y$, namely $p(\theta,y)=p(y\mid\theta)\,p(\theta)$. The posterior is the conditional distribution of $\theta$ given $y$ under this joint distribution. Bayes’ theorem is simply the rule for computing it: $$\tag{1.1} p(\theta\mid y) = \frac{p(y\mid\theta)\,p(\theta)}{p(y)} = \frac{p(y\mid\theta)\,p(\theta)}{\int_\Theta p(y\mid\tilde\theta)\,p(\tilde\theta)\,\mathrm{d}\tilde\theta}.$$ (If $\theta$ is discrete, the integral is replaced by a sum.)

The denominator $p(y)$ deserves a comment. It is obtained by averaging the probability of the observed data over all possible values of the parameter, weighted by the prior. It is therefore the probability of what was actually observed, computed before the data were seen, and it is called the marginal likelihood or prior predictive probability of the data. It does not depend on $\theta$, and this is what matters for calculation. Its only role in (1.1) is to make the posterior integrate to one. For this reason we often write Bayes’ theorem as a proportionality, $$p(\theta\mid y) \propto p(y\mid\theta)\,p(\theta),$$ or, in words, posterior is proportional to likelihood times prior. The prior and the information from the data combine by multiplication, and the product is then rescaled to integrate to one. A value of $\theta$ is favoured by the posterior if it was plausible beforehand and if it makes the observed data probable; a value that fails on either count is discounted.

The marginal likelihood will play little part in this half of the module, but it becomes important when competing models are to be compared, since it measures how well each model predicted the data before they were seen.

This gives us the central idea of the course. Bayesian statistics describes how an idealised agent ought to learn from data, provided two conditions hold. First, the agent’s knowledge must be expressible, at least approximately, as a probability distribution. Second, the agent must be able to carry out the necessary calculations, and in particular to evaluate the integral in (1.1). Neither condition is trivial. The first raises questions about where priors come from, which we take up in Chapter 10. The second is a matter of computation, and a large part of the course, from Chapter 8 onwards, is concerned with it.

1.3 A diagnostic test

The following example is simple, but it displays in miniature most of what Bayesian reasoning involves, and it shows how easily intuition can go astray without it.

A doctor is concerned that a patient may have a rare but serious disease. Let $\theta=1$ if the patient has the disease and $\theta=0$ otherwise, and let $y=1$ if a diagnostic test returns a positive result and $y=0$ otherwise. Here both the parameter and the data are binary.

1.4 Objections to Bayesian inference

For much of the twentieth century there was a divide, at times a hostile one, between frequentist and Bayesian statisticians. It is worth knowing the main lines of the argument, both because they clarify what Bayesian methods claim to do and because you will meet them in practice.

The principal frequentist objection is that Bayesian inference is not objective. Science, on this view, is meant to reach conclusions that any competent investigator would reach from the same evidence. If the conclusions of an analysis depend on the prior beliefs of the analyst, then two scientists looking at the same data may disagree, and the conclusions seem to say as much about the scientists as about the world.

The Bayesian reply comes in several parts. The first is that data acquire meaning only by changing what we believe; a set of numbers without any context about what they measure or what was expected tells us nothing. The second is that frequentist methods also rest on choices and assumptions (of model, of test statistic, of significance level, of stopping rule) and that these play a role similar to that of a prior, though less openly. Some Bayesians go further and argue that frequentist procedures, where they are sensible, are Bayesian procedures with an unstated prior.

A Bayesian may respond to the charge of subjectivity in either of two ways. One is to accept it. On this view Bayesian inference is a tool for an individual, with genuinely held beliefs, to learn from data and to make decisions, and there is nothing to apologise for in that. The Decision Theory half of the module develops this view further. The other is to make statements that do not depend on any one person’s beliefs. For example, one can say: these data are sufficient to move anyone who held a prior of such a form to a posterior of such a form. The information in the data is then measured by its capacity to change the minds of a range of hypothetical people. Reporting how conclusions vary as the prior is varied, known as a sensitivity analysis, is a practical form of the same idea. And, as we shall see repeatedly, when the data are plentiful the posterior depends very little on the prior, so that people who begin with different beliefs are brought into agreement by the evidence.

2.1 Sample spaces, events and probabilities

A sample space $\Omega$ is a set of possible outcomes, or states of the world. One can think of it as an ‘atomic’ description: each outcome specifies everything that is relevant to the problem at hand. For a single roll of a die, $\Omega=\{1,2,3,4,5,6\}$. For a patient undergoing a test, $\Omega$ might consist of the four combinations of disease status and test result.

An event $E\subseteq\Omega$ is a set of outcomes, and we say that $E$ occurs if the outcome that is realised belongs to it. For the die, ‘an even number is rolled’ is the event $\{2,4,6\}$. An event space $F$ is a collection of events to which probabilities will be assigned. One can think of it as describing what we are able to observe or ask about. (For technical reasons, when $\Omega$ is uncountable, it is not always possible to assign probabilities consistently to every subset, and $F$ must be restricted to a suitable collection called a $\sigma$-algebra. We shall not need these details.)

A probability function $P$ assigns a number $P(E)$ to each event $E$ in $F$.

2.2 Axioms

Kolmogorov, in 1933, gave the axioms that now form the standard basis of probability theory. A triple $(\Omega,F,P)$ is called a probability space if

1. $P(E)\ge0$ for all $E\in F$;

2. $P(\Omega)=1$;

3. $P\big(\bigcup_{i=1}^\infty E_i\big)=\sum_{i=1}^\infty P(E_i)$ whenever the events $E_1,E_2,\ldots$ are pairwise disjoint, that is, $E_i\cap E_j=\emptyset$ for $i\ne j$.

Here $E_i\cup E_j$ is the event that $E_i$ or $E_j$ (or both) occurs, $E_i\cap E_j$ the event that both occur, and $\emptyset$ the empty set. In Kolmogorov’s system, conditional probability is then defined by $$P(E_i\mid E_j) = \frac{P(E_i\cap E_j)}{P(E_j)}, \qquad \text{provided } P(E_j)>0.$$

Kolmogorov’s axioms say nothing about what probability means. They are equally at home with frequencies and with degrees of belief. Bayesians have therefore asked a different question: what properties ought a person’s degrees of belief to have if they are to be coherent? Hoff, following de Finetti and most Bayesian authors, lists the following:

1. $0\le P(E)\le1$ for all $E$;

2. $P(E_i\cup E_j)=P(E_i)+P(E_j)$ if $E_i\cap E_j=\emptyset$;

3. $P(E_i\cap E_j)=P(E_i\mid E_j)\,P(E_j)$.

In this system conditional probability is a primitive notion. $P(E_i\mid E_j)$ is the degree of belief one would have in $E_i$ on learning that $E_j$ is true, and Axiom P3 is a requirement that such conditional beliefs fit together consistently with unconditional ones. The conclusion is that degrees of belief should obey the same rules as Kolmogorov’s probabilities, so that the whole of probability theory is available to describe them. (The one difference, that de Finetti’s system requires additivity only for finitely many events rather than countably many, will not concern us.)

Why should degrees of belief satisfy these rules? One well-known argument, due to de Finetti, uses betting. Suppose your degree of belief in an event $E$ is the price $P(E)$ at which you would be willing either to buy or to sell a ticket that pays £1 if $E$ occurs and nothing otherwise. If your prices violate the axioms, then an opponent can construct a collection of bets, each of which you regard as fair, that together lose you money whatever happens. Such a collection is called a Dutch book. For example, if you set $P(E)=0.6$ and $P(E^c)=0.6$, violating P2 (since $E$ and its complement are disjoint with union $\Omega$), the opponent sells you both tickets for £1.20; exactly one of them pays out, so you receive £1 and lose 20 pence for certain. Beliefs that cannot be exploited in this way are called coherent, and it can be shown that coherence is equivalent to satisfying the axioms. Arguments of this kind are developed further in the Decision Theory half of the module.

2.3 Partitions and Bayes’ rule

Many calculations involve dividing the sample space into a number of mutually exclusive possibilities.

In other words, exactly one of the $H_k$ is true whenever $H$ is. In the diagnostic example, the events ‘disease present’ and ‘disease absent’ form a partition of $\Omega$.

Suppose that $\{H_1,H_2,\ldots\}$ is a partition of $\Omega$, so that $\sum_kP(H_k)=1$, and let $E$ be any event with $P(E)>0$. Two results follow directly from the axioms.

• The law of total probability, or marginal probability rule: $$P(E) = \sum_k P(E\cap H_k) = \sum_k P(E\mid H_k)\,P(H_k).$$ The first equality holds because the events $E\cap H_k$ partition $E$; the second is Axiom P3 applied to each term. In words, the probability of $E$ is a weighted average of its conditional probabilities under each hypothesis, weighted by the probabilities of the hypotheses.

• Bayes’ rule: $$P(H_j\mid E) = \frac{P(E\mid H_j)\,P(H_j)}{P(E)} = \frac{P(E\mid H_j)\,P(H_j)}{\sum_k P(E\mid H_k)\,P(H_k)}.$$ This follows by writing $P(H_j\cap E)$ in two ways using Axiom P3, $P(H_j\mid E)P(E)=P(E\mid H_j)P(H_j)$, and then using the law of total probability for the denominator.

The statistical reading of Bayes’ rule is the one we met in Chapter 1. Think of the $H_k$ as competing hypotheses about the state of the world, perhaps corresponding to different values of a parameter, and of $E$ as the event that an experiment produces a particular result. Bayes’ rule converts the probabilities $P(E\mid H_k)$ of the result under each hypothesis, which are usually what a scientific model provides, into the probabilities $P(H_k\mid E)$ of the hypotheses given the result, which are what we actually want to know.

2.4 Independence and conditional independence

Events $F$ and $G$ are independent if $$P(F\cap G) = P(F)\,P(G),$$ and they are conditionally independent given $H$ if $$P(F\cap G\mid H) = P(F\mid H)\,P(G\mid H).$$ Axiom P3, applied conditionally on $H$, gives $P(F\cap G\mid H)=P(G\mid H)\,P(F\mid H\cap G)$. Comparing this with the definition shows that, provided $P(G\mid H)>0$, conditional independence of $F$ and $G$ given $H$ is equivalent to $$P(F\mid H\cap G) = P(F\mid H).$$ This is the more intuitive form: if we already know that $H$ is true, then learning that $G$ is also true does not change our belief about $F$.

Independence and conditional independence are different properties, and neither implies the other. The distinction is central to this course, and the diagnostic example illustrates it well. Suppose the test is applied twice, and let $F$ and $G$ be the events that the first and second results are positive. Given the patient’s disease status, the two results are independent, since each depends only on the behaviour of the test. But before we know the disease status, they are not independent. A positive first result makes disease more likely, and so makes a positive second result more likely too. Indeed, using the numbers from Chapter 1, $$\begin{aligned} P(G) &= 0.95\times0.001+0.02\times0.999\approx0.021,\\ P(G\mid F) &= \frac{P(F\cap G)}{P(F)} = \frac{0.95^2\times0.001+0.02^2\times0.999}{0.021}\approx0.062, \end{aligned}$$ so learning $F$ roughly triples the probability of $G$. The dependence arises entirely through the unknown disease status.

Almost every model in this course has this structure. The observations are modelled as independent given the value of a parameter, but since the parameter is unknown the observations are not independent unconditionally. It is precisely this dependence that allows us to learn: each observation tells us something about the parameter, and so something about the observations yet to come. Chapter 3 makes this idea precise.

2.5 Random variables

To a Bayesian, any unknown quantity may be treated as a random variable, and its probability distribution describes our uncertainty about it. We distinguish two main kinds.

2.6 Transformations

We shall often need the distribution of a function of a random variable: of $1/\theta$ when we know the distribution of $\theta$, for example. For discrete variables this is a matter of adding up probabilities. For continuous variables the densities must be adjusted.

Let $Y$ be continuous with density $p_Y$, and let $W=f(Y)$, where $f$ is one-to-one and continuously differentiable. Then $W$ has density $$p_W(w) = p_Y\big(f^{-1}(w)\big)\left|\frac{\,\mathrm{d}y}{\,\mathrm{d}w}\right|,$$ where $y=f^{-1}(w)$. To see why, suppose $f$ is increasing. Then $P(W\le w)=P(Y\le f^{-1}(w))=F_Y(f^{-1}(w))$, and differentiating with respect to $w$ by the chain rule gives the formula. If $f$ is decreasing, the same argument gives the formula with a minus sign, which the absolute value takes care of. The factor $|\,\mathrm{d}y/\,\mathrm{d}w|$ records how the transformation stretches or compresses the axis: a small interval of length $\delta$ in $w$ corresponds to an interval of length about $|\,\mathrm{d}y/\,\mathrm{d}w|\,\delta$ in $y$, and the two intervals must carry the same probability.

The same reasoning extends to random vectors. If $\boldsymbol W=f(\boldsymbol Y)$ with $f$ one-to-one and differentiable, then $$p_{\boldsymbol W}(\boldsymbol w) = p_{\boldsymbol Y}\big(f^{-1}(\boldsymbol w)\big)\,\big|\det J\big|,$$ where $J$ is the Jacobian matrix of the inverse transformation, with entries $J_{ij}=\partial y_i/\partial w_j$. The absolute determinant plays the part of $|\,\mathrm{d}y/\,\mathrm{d}w|$, measuring how the transformation changes small volumes.

These formulae have a consequence that will matter when we discuss the choice of prior. A density on $\theta$, and the density it induces on a transformed parameter $\phi=g(\theta)$, do not in general have the same shape. In particular, a prior that is flat (uniform) in $\theta$ is not flat in $\log\theta$ or in $1/\theta$. So the statement ‘I have no preference among the possible values of the parameter’ is ambiguous until we say which parameterisation we mean. We return to this in Chapter 10.

3.1 Motivation

Roll a fair die $n$ times, and let $Y_i=1$ if the $i$th roll shows a six and $Y_i=0$ otherwise. The rolls are independent, and each shows a six with probability $1/6$, so the probability of any particular sequence of results $(y_1,\ldots,y_n)$ is $$p(y_1,\ldots,y_n) = \prod_{i=1}^n\left(\tfrac16\right)^{y_i}\left(\tfrac56\right)^{1-y_i} = \left(\tfrac16\right)^{y}\left(\tfrac56\right)^{n-y}, \qquad y=\sum_{i=1}^n y_i.$$ This probability depends only on the number of sixes, $y$, and the number of other results, $n-y$. Every sequence with the same number of sixes has the same probability, whatever the order in which the sixes appear. The sequence $(1,0,0)$ is exactly as probable as $(0,0,1)$.

This symmetry under reordering turns out to be the essential property, and it holds much more widely than for independent rolls of a fair die.

3.2 Definition

To judge a set of quantities exchangeable is to judge that their labels carry no information: knowing which observation is which would not change your beliefs about the collection. This is often a reasonable judgement. If the $Y_i$ are measurements on individuals drawn at random from a population, and there is nothing else to distinguish one individual from another, then we have no reason to expect the first measurement to differ systematically from the fifth. It is not always reasonable. Measurements taken in time order, where later values may depend on earlier ones, are not usually exchangeable. Nor are measurements on individuals known to belong to different groups, such as patients given different treatments, though they may be exchangeable within each group.

Independent and identically distributed (i.i.d.) random variables are exchangeable, since $$p(y_1,\ldots,y_n) = \prod_{i=1}^n p(y_i) = \prod_{i=1}^n p(y_{\pi_i}) = p(y_{\pi_1},\ldots,y_{\pi_n}),$$ the middle equality holding because a product does not depend on the order of its factors. The same applies to infinite i.i.d. sequences.

The converse is false: exchangeable variables need not be independent. The next theorem shows the most important way in which exchangeable but dependent variables arise.

Such variables are, however, not independent marginally, for the reason explained in the previous chapter. Observing $Y_1$ changes our beliefs about $\theta$, and hence about $Y_2$. It is exactly this dependence that makes learning possible. If the observations were marginally independent, no number of them would tell us anything about the next.

3.3 De Finetti’s theorem

Theorem 3.2 says that the familiar model structure, conditionally i.i.d. observations with a prior on the parameter, always produces exchangeable observations. The converse also holds, provided the sequence can be extended indefinitely.

Combining the two theorems, $$\left.\begin{array}{l} Y_1,\ldots,Y_n\mid\theta \text{ are i.i.d.}\\ \theta\sim p(\theta)\end{array}\right\} \iff Y_1,\ldots,Y_n \text{ are exchangeable for all } n.$$

We do not prove the theorem here. For binary sequences a fairly elementary proof is given in Bernardo and Smith, §4.3. The binary case also shows what the parameter is. If $Y_1,Y_2,\ldots$ is an exchangeable sequence of zeros and ones, then the proportion of ones among the first $n$, $\bar Y_n=\frac1n\sum_{i=1}^nY_i$, converges as $n\to\infty$, and the limit is the parameter $\theta$. The prior $p(\theta)$ is therefore our belief about the long-run proportion of ones in the sequence, and given that proportion the observations behave like independent Bernoulli trials with that probability of success.

It is worth pausing on why this matters. A person who judges only that the labels on a sequence of observations are irrelevant, and who is willing to imagine the sequence continuing indefinitely, is thereby committed to modelling the observations as if they were i.i.d. given some parameter, with a prior distribution on that parameter. The parameter and its prior are not extra assumptions laid on top of the data. They arise from a judgement of symmetry about quantities that can actually be observed. De Finetti held that probability statements should ultimately concern observable quantities, and his theorem explains how parameters, which are not directly observable, nonetheless earn their place in a model. For a modern account that places prediction at the centre of Bayesian modelling, see Fortini and Petrone (2025), Statistical Science 40, 40–67.

The theorem does not tell us which sampling model or prior to use. That remains a matter of judgement, informed by what we know of the problem. It tells us only the form the model must take.

3.4 Finite populations

De Finetti’s theorem requires an infinite exchangeable sequence, and the requirement is not a technicality. Suppose we draw individuals at random, without replacement, from a finite population of size $N$. The resulting observations are exchangeable, since the order of drawing is irrelevant, but they are not in general conditionally i.i.d. given any parameter. Exercise 3.1 gives an extreme example. The reason is that drawing without replacement induces negative dependence (drawing one individual with a property makes the next less likely to have it), whereas conditionally i.i.d. observations can only be positively dependent or independent.

When the sample size $n$ is small compared with the population size $N$, however, the effect of not replacing is negligible, and we may treat the observations as approximately conditionally i.i.d. This is the situation in most applications, and it is the justification for the models used in the rest of the course. Most of the models we consider assume that the data are conditionally i.i.d. given some parameter $\theta$, which may be a vector.

4.1 Setting

A familiar kind of question runs as follows. We draw $n$ individuals at random from a population and find that some of them have a certain property. What does this tell us about the proportion of the whole population that has the property? Some examples:

• In a clinical trial of a new treatment for psoriasis, 12 of 15 patients showed a marked improvement compared with the standard treatment. What is the probability that a future patient will improve?

• On 5 September 2014, a YouGov poll of 1084 voters on the Scottish independence referendum found 51% for Yes and 49% for No, excluding those who would not vote or did not know. What is the probability that Yes was ahead in the population?

• Of 12 lecturers in a small department at a UK university, 3 are foreign nationals.

• Of 25 items selected from a day’s production for quality control, none is defective. Does that mean the production process makes no defective items?

To set up a model, label the individuals in the population $i=1,\ldots,N$ and let $$Y_i=\begin{cases}1 & \text{if individual } i \text{ has the property},\\ 0 & \text{otherwise}.\end{cases}$$ The quantity of interest is the population proportion $\theta=\sum_{i=1}^NY_i/N$. We observe $Y_i$ for a sample of $n$ individuals, usually with $n$ much smaller than $N$, and ask how these observations should change our beliefs about $\theta$.

If we have no information that distinguishes one sampled individual from another, it is natural to judge $Y_1,\ldots,Y_n$ exchangeable. Since $N$ is large compared with $n$, the discussion of Chapter 3 then justifies the model $$Y_1,\ldots,Y_n\mid\theta \stackrel{\text{iid}}{\sim}\operatorname{Bernoulli}(\theta),$$ in which each observation is a one with probability $\theta$, independently of the others, given $\theta$.

4.2 Likelihood and sufficiency

Under this model the probability of the observed sequence is $$p(y_1,\ldots,y_n\mid\theta) = \prod_{i=1}^n\theta^{y_i}(1-\theta)^{1-y_i} = \theta^{y}(1-\theta)^{n-y}, \qquad\text{where } y=\sum_{i=1}^n y_i.$$ Notice that the likelihood depends on the data only through $y$, the total number of ones. Two samples of the same size with the same number of ones give exactly the same likelihood function, and so, whatever the prior, the same posterior. We express this by saying that $Y=\sum_iY_i$ is a sufficient statistic for $\theta$: once we know $y$, the individual values $y_i$, and in particular the order in which the ones and zeros occurred, carry no further information about $\theta$.

The total $Y$ has a binomial distribution, $$p(y\mid\theta) = \binom ny\theta^y(1-\theta)^{n-y}, \qquad y=0,1,\ldots,n.$$ This differs from the likelihood of the full sequence only by the factor $\binom ny$, which does not involve $\theta$. Since Bayes’ theorem determines the posterior only up to a constant factor, it makes no difference whether we work with the sequence or with the total. We shall move freely between the two.

4.3 A uniform prior

Next we need a prior. Suppose, to begin with, that we have no particular information about $\theta$, and that we regard all values between $0$ and $1$ as equally plausible. This suggests the uniform prior $\theta\sim\operatorname{Unif}(0,1)$, with density $$p(\theta) = 1, \qquad 0<\theta<1,$$ under which any two intervals of the same length have the same prior probability. (We shall see in Chapter 10 that ‘equally plausible’ is less straightforward than it sounds, but the uniform prior is a sensible starting point.)

The posterior is then $$p(\theta\mid y) \propto p(y\mid\theta)\,p(\theta) = \theta^y(1-\theta)^{n-y}, \qquad 0<\theta<1.$$ As a function of $\theta$, this has the form of the density of a Beta distribution, with its normalising constant omitted.

The Beta family is very flexible on the interval $(0,1)$. With $a=b=1$ it is uniform. With $a=b>1$ it is symmetric about $\frac12$ and peaked, more sharply as $a$ and $b$ increase. With $a>b$ it is skewed towards one, and with $a<b$ towards zero. With $a$ or $b$ below one the density is unbounded at the corresponding end of the interval. The form of the variance shows that, for a fixed mean, the distribution becomes more concentrated as $a+b$ increases.

Comparing the posterior with the Beta density, we conclude that $$\theta\mid y\sim\operatorname{Beta}(y+1,\;n-y+1).$$ For the psoriasis trial, with $y=12$ and $n=15$, the posterior is $\operatorname{Beta}(13,4)$. Its mean is $13/17\approx0.76$, and the posterior probability that more than half of future patients would improve is $P(\theta>0.5\mid y)\approx0.99$.

4.4 The kernel method

The step we have just taken will recur throughout the course, and it is worth setting it out explicitly.

1. Write down the posterior up to a constant, $p(\theta\mid y)\propto p(y\mid\theta)\,p(\theta)$. At each stage, discard any factor that does not involve $\theta$.

2. Recognise the result as the kernel of a known distribution, that is, as its density with the normalising constant removed.

3. Conclude that the posterior is that distribution.

The argument is valid because a probability density is determined by its kernel. If two densities are proportional to one another, then, since both integrate to one, the constant of proportionality must be one and the densities are equal. The practical benefit is considerable. It spares us the integral in the denominator of Bayes’ theorem, which is the hardest part of the calculation, and it gives us the posterior in a form whose properties (mean, variance, quantiles) are already known.

4.5 Prediction

Often the most useful thing a posterior distribution can do is predict future observations. What should we expect of the next individual we sample? By the tower property (2.1), conditioning on $\theta$, $$\operatorname{E}[Y_{n+1}\mid y_1,\ldots,y_n] = \int_0^1 \operatorname{E}[Y_{n+1}\mid\theta,y_1,\ldots,y_n]\,p(\theta\mid y)\,\mathrm{d}\theta.$$ Given $\theta$, the new observation is independent of the old ones and has mean $\theta$, so $\operatorname{E}[Y_{n+1}\mid\theta,y_1,\ldots,y_n]=\theta$, and $$\operatorname{E}[Y_{n+1}\mid y_1,\ldots,y_n] = \int_0^1\theta\,p(\theta\mid y)\,\mathrm{d}\theta = \operatorname{E}[\theta\mid y] = \frac{y+1}{n+2}.$$ Since $Y_{n+1}$ takes only the values $0$ and $1$, this is also the probability that the next individual has the property. The prediction does not use any single ‘best’ value of $\theta$. It averages over all values of $\theta$, weighted by their posterior probability.

4.6 Beta priors

The uniform distribution is the special case $\operatorname{Beta}(1,1)$ of the Beta family. We now show that the calculation of the previous sections goes through for any Beta prior. Suppose $\theta\sim\operatorname{Beta}(a,b)$ and $Y\mid\theta\sim\operatorname{Bin}(n,\theta)$. Then $$\begin{aligned} p(\theta\mid y) &\propto p(\theta)\,p(y\mid\theta) = \frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)}\theta^{a-1}(1-\theta)^{b-1}\times\binom ny\theta^y(1-\theta)^{n-y}\\ &\propto \theta^{a+y-1}(1-\theta)^{b+n-y-1}, \end{aligned}$$ and so, by the kernel method, $$\theta\mid y\sim\operatorname{Beta}(a+y,\;b+n-y).$$ The updating rule is simple: add the number of successes to $a$, and the number of failures to $b$.

This suggests a useful way to think about the prior. The $\operatorname{Beta}(a,b)$ prior behaves exactly as though we had begun with a uniform prior and had already observed $a-1$ successes and $b-1$ failures. Alternatively, and more commonly, one says that the prior carries the information of a sample of size $a+b$ in which the proportion of successes is $a/(a+b)$. The quantity $a+b$ is often called the prior sample size. This interpretation helps in choosing a prior: we can set the prior mean $a/(a+b)$ to our best guess for $\theta$, and then choose $a+b$ to reflect how many observations we think our prior knowledge is worth.

4.7 Conjugacy

We have seen that a Beta prior combined with binomial data yields a Beta posterior. This property has a name.

The class of Beta distributions is thus conjugate for binomial data. Conjugate priors have obvious advantages. Computing the posterior amounts to updating a few parameters, and the properties of the posterior are those of a familiar distribution. They also make the contribution of the prior easy to interpret, as with the prior sample size above.

Their disadvantage is that a conjugate family may not be flexible enough to represent what we actually believe. A Beta density has at most one mode, for example, and cannot express the belief that $\theta$ is probably either near $0.2$ or near $0.7$. A useful compromise is a mixture of conjugate priors. If the prior is a mixture of Beta distributions, $p(\theta)=\sum_kw_k\,\operatorname{Beta}(\theta\mid a_k,b_k)$, then the posterior is also a mixture of Beta distributions, each component updated in the usual way, with the mixture weights themselves updated in proportion to how well each component predicted the data. Any continuous prior on $(0,1)$ can be approximated arbitrarily well by such a mixture.

In Chapter 10 we shall see that conjugate priors exist for a whole class of models, the exponential families, of which the binomial, Poisson, exponential and normal models are all members.