This question came from Andrew Wills.
He and his colleagues had submitted a paper to a journal quantifying the intra- and inter-observer reliability of a new tool to measure range of movement of a joint. They used the Limits of Agreement method as in Bland and Alltman (1986).
One of the editors commented that, in one of the difference vs mean plots, all of the differences in observations were contained within the 95% limits. He suggested that this cannot be correct if 5% of observations should fall outside the limits.
The reliability study was performed on 17 subjects.
I replied as follows.
If you have 17 observations, 5% is less than one observation. Why should we expect to see any observations outside the limits? In statistical terms, the number outside the limits would follow a Binomial distribution with parameters n = 17 and p = 0.05. The probablity of 0 for this distribution is 0.050 * (1-0.05)17 = 0.9517 = 0.42. This is quite a substantial probablity. For nearly half of the studies we did with 17 subjects all observations would be within limits.
In general, for larger n, in 95% of plots we would get between
0.05*n - 1.96*root(0.05*(1-0.05)n) and 0.05*n + 1.96*root(0.05*(1-0.05)n),
i.e. between
0.05*n - 0.43*root(n) and 0.05*n + 0.43*root(n)
observations outside the limits of agreement. If the lower figure here is negative, then n is too small for the approximation.
For 100 observations, we expect to get between 5 - 4.3 and 5 + 4.3, i.e. between 0.7 observations and 9.3 observations outside the 95% limits of agreement. We would therefore be a little surprised to get none at all, but not to get only one.
Looking at it another way, we can find the sample size so that the probability of no observed differences lying outside the limits of agreement is any chosen value. For example, the sample size so that the probability is 1 in 20 or 0.05 that all observed differences wilol be contained within the limits is given by 0.95n = 0.05. This is n = 58.4. So if we have 58 observations or fewer, the probability that there will be no observed difference outside the 95% limits of agreement is greater than one in twenty.
The mean and variance of differences which we calculate are estimates of the mean and variance in the population from which the sample comes. The estimates of the limits of agreement calculated using them are estimates of the limits which apply to the population which the sample represents, not for that particular sample. We should not expect 5% of observations in the sample to fall outside them.
Back to measurement studies menu.
Back to Martin Bland's home page.
To Douglas Altman's home page.
This page maintained by Martin Bland.
Last updated: 29 September, 2004.