How to Upset the Statistical Referee

Martin Bland,
Dept of Health Sciences,
University of York

Talk first presented to the London Hypertension Society.


I was first asked to speak on this topic by Donald Singer for a meeting organised by the London Hypertension Society. The aim, of course, is not to upset the statistical referee, but this way round is more fun.

Researchers come to me with comments from statistical referees quite often. I usually agree with the referee. This is not as bad as it sounds, because I can often show the frustrated authors how to do what the referee suggests and so get their papers accepted. We must accept, however, that referees, statistical or otherwise, are fallible people just like you and me and, like us, they get it wrong. After all, the authors might have spent months working on their paper and the referee is unlikely to spend more than half a day on it. Sometimes I'll disagree with the referee and help the author to fight, but this is definitely the minority of cases. And, of course, those referees (of whom there are far too many) who recommend changing or rejecting my work are fools and charlatans.

When I am the referee, on the other hand, I find it is the authors who are unfit to be let out alone. I find myself gasping at the folly of my fellow men and women and racing down the corridor to show my colleagues the latest jaw-dropper. I could not resist, for example, the grant applicant who asked under computing for money to buy 'soft wear'; a nice cashmere sweater for cold computer rooms, perhaps. I have often thought it a pity that such things do not get to a wider audience. Accordingly, when I was given this tempting title I decided to make it is a personal account and use some of my referee's reports. I based it on my experience as a statistical referee for the Lancet, as summer relief in 1994 and 1995. I doubt very much that things have improved dramatically since then, but if you know different, let me know.

In what follows, I shall use quotes from some of my reports to the Lancet. As all the papers were confidential, I have changed a few details to protect the ignorant. As a rule, I have no qualms about publicly pointing out the mistakes of others once they have been published. If we do not do this, the conclusions that follow from these mistakes will be quoted by others, usually without any criticism, and become generally accepted. Also, if you publish your work, you must be prepared to defend your position, or amend it. But when work has not been published, but rather is at some point on the twisting road to publication, I think that it would be unfair to criticise it publicly. On the other hand, such work can often be very illuminating. I'm not the only statistician who has thought that I would really like this paper that I am reviewing to be published, because it would make a wonderful teaching example of what not to do. I have therefore done my best in what follows to describe real papers but at the same time to preserve the confidentiality of the reviewing process. I have disguised the nature of the research, sometimes calling the variables 'X' rather than giving them their proper name. I have even changed the some of the numbers. However, I do not think that I have changed or exaggerated the nature of the statistical mistake that I was pointing out. These quotes come from reports on just 15 papers, so if something comes up several times it may be pretty common.

The Allstat Sample

After I had given the talk a couple of times, I wanted generalise a bit and incorporate the views of other statistical referees. I used a completely non-statistical approach: a convenience sample with a low response rate. I used Allstat, an email list that keeps statisticians in touch with one another. I broadcast the following message:

Subject: Statistical referees for medical journals

To allstaters who act as statistical referees for medical journals.

I am preparing a talk entitled "How to upset the statistical referee". This is based on my own (rather limited) adventures with the Lancet. I wondered what are the pet hates of other referees, the things which really irritate them? If there is something which authors do which really upsets you, could you tell me what it is? I shall, of course, post a summary of replies.

Allstat responded beyond all expectations. I received 35 replies, many of which were very extensive and wide-ranging. I found this rather overwhelming and I never did produce that summary of replies which I had promised. Apologies to Allstat for that.

Eventually, I managed to sort and classify these replies. I have added the Allstat comments wherever they fit in with my own and added them separately where they do no not.

I think of this as my only purely qualitative research project, using two convenience samples (of reviews and of respondents to my Allstat message) one of which was self-selected, used to triangulate the theory generated.

You may be surprised by some of the things which my colleagues and I object to, as you will see many of them appearing frequently in journals. Some of them are what might be termed 'parastatistics', statistics as practiced by users of statistics but not by statisticians. Not all statisticians would agree with me or with my respondents, either, and we should not forget that the collective noun for statisticians is a 'variance'. Given these cautions, I hope that what follows will give a good introduction as to what might going through the mind of the statistical referee for your paper. Here we go.

Effects which are not significant

My most frequent and severe complaints concern significance tests and confidence intervals. I think that one of the greatest statistical crimes is to carry out a significance test, get a large P value, and then interpret this as meaning that there is no difference. This happens again and again. My comments included:

`This is a small trial of two similar regimes. They interpret "no significant difference" as meaning "no difference". I do not think that there was any chance of a significant difference anyway. They should present confidence intervals as in the Lancet's guidelines.'

`Not significant should NOT be interpreted as "no change".'

`The conclusion interprets "not significant" as meaning "no difference", which it does not. It means that a difference has not been shown to exist.'

`The habit of reporting non-significant differences as no differences gives me no confidence in the report of no change here. I suggest that some data be included.'

A couple of my Allstat respondents mentioned this, too:

'Interpreting P>0.5 as "evidence" of no difference, without reference to sample size or confidence intervals'

'Interpreting non-significance as "no difference" to such an extent that the Discussion focuses around why this should be also grates high on the pet hates scale.'

Lack of confidence intervals

Wherever possible, authors should report confidence intervals for differences, not just significance tests. For years statisticians have been trying to persuade researchers of this (e.g. Gardner and Altman 1986). This is the usual guideline of most journals anyway, including the Lancet. The current guideline, from the Lancet website, include: 'When possible, quantify findings and present them with appropriate indicators of measurement error or uncertainty (such as confidence intervals). Avoid relying solely on statistical hypothesis testing, such as the use of P values, which fails to convey important quantitative information.' Authors continually ignore this, and my papers were no exception. My comments included:

`The results should be presented as confidence intervals, not significance tests. For example, the non-significant 19% adverse reactions on the test treatment compared to 12% on the standard treatment is a relative risk of adverse reaction 1.5, 95% confidence interval 0.5 to 4.6. Thus the data are compatible with more than four times as many adverse reactions on the new than on the standard treatment. For the presence of X, one in each group, the relative risk is 1.3; the 95% confidence interval is 0.08 to 20. Thus the data are compatible with more than twenty times as many Xs on the new than on the standard treatment!'

`A confidence interval for the mean difference would be much better than significance tests. A non-significant difference in 10 subjects cannot be interpreted.'

`A finding of "not significant" is meaningless in 4 or 5 subjects. Confidence intervals should be used.'

`This non-significant difference, reported as "unchanged", is proportionately greater than many significant differences in this paper. A confidence interval for the mean difference would be much better.'

My Allstat sample agreed with me, four respondents mentioning the point. Typical comments were:

'Papers where the only statistics are p-values.'

'Insisting on giving the test statistics, and refusing to give estimated effects.'

Presenting P values

Computers now print out the exact P values for most test statistics. These should be given, rather than change them to "not significant" or P>0.05. Similarly, if we have P=0.0072, we are wasting information if we report this as P<0.01. This method of presentation arises from the pre-computer era, when calculations were done by hand and P values had to be found from tables. Personally, I would quote this to one significant figure, as P=0.007, as figures after the first do not add much, but the first figure can be quite informative. Two of my 15 Lancet papers had these problems:

`A report of "p=NS" is not very informative. If significance tests must be used, the exact P value is preferable.'

`These are P=0.01, P=0.006, P=0.05, not P<0.01, P<0.006, P<0.05. In fact, the first is actually P=0.012, so what they have written is incorrect.'

Several Allstat respondents raised this issue. Their comments included:

'Using 'NS' for any p>0.05, including p=0.0501 (three replies made this point)'

'Showing a table of p-values to huge numbers of decimal places when they're significant, but not even to one place when not: 'NS' should be banished!'

'Also, statistical methods sections which say "all results were regarded as significant at the 5% level", followed by results where p<0.05 or p=NS.'

'The term "failed to achieve statistical significance"'.

'I mainly derive irritation from little things, such as "P<0.013"'

So if you want to avoid irritating the statistical referee (and you may not) you should quote your P values correctly to one significant figure.

More on P values

P values greatly exercised my Allstat respondents. Three complained about multiple testing:

'Carrying out hundreds of significance tests, instead of either addressing specified hypotheses, or admitting that the study is descriptive.'

'Massed p-values, like firing a blunderbuss into a fishpond.'

'Skipping of a non-significant finding on the principal outcome to concentrate on a significant result in a side issue, whether this is the infamous sub-group or some minor outcome measure.'

If we carry out many tests of significance, even if the null hypotheses are all true #we expect that 5% of them will be significant. If we then concentrate on these significant tests in our report we can give a very misleading impression. One of my favourite examples is due to Newnham et al. (1993), who randomized pregnant women to receive a series of Doppler ultrasound blood flow measurements or to control. They found a significantly higher proportion of birthweights below the 10th and 3rd centiles in the Doppler group compared to the controls (P=0.006 and P=0.02). These were only two of many comparisons and at least 35 were reported in the paper. Only these two were reported in the abstract. (Birthweight was not the intended outcome variable for the trial). This trial was widely reported and the finding that Doppler ultrasound reduced birthweight was reported in the national news.

Another Allstat respondent raised an interesting point:

'If you aren't already a fan you should be watching ER. Last night there was talk of a chi-squared analysis showing significance at the 0.06 level, "so we only need one more positive result"'.

I wonder how many of the television audience understood that one. Most statistical analyses assume that the observations are independent of one another. If we do not have independent observations, an analysis which requires this will be wrong. If we test each time an observation is added, the observations cannot be independent, because an observation will only be made if the previous ones did not show a significant difference. We would be doing multiple testing, and the probability of a test reaching the nominal P value of 5% if the null hypothesis were true would be much more than 5%. I doubt that people who do this would actually mention it in their paper. The final test would be presented as if it were the only one carried out. Doing this could be the result of ignorance, researchers genuinely thinking that this is a valid procedure. If the researcher knows that the procedure is not valid, it is fraud. In either case, we would end with a potentially false and misleading conclusion.

I don't know quite what an Allstat respondent meant by this complaint:

'Authors who use p-value cut-offs other than <0.05, <0.01 or <0.001 and then don't attempt to justify the levels they use (I find this especially in the papers concerning large animals where there are only 3 cows and insufficient data for any conventional statistical significance at all).'

I suspect that he or she was referring to authors who regard differences as significant if P<0.10 or even higher probabilities. This can be justifiable in some circumstances. An example might be in the screening of novel chemicals for pharmaceutical activity. We put all chemicals through an initial screen intended to select some for a further more intensive screen. It is more important to detect any which have biological activity than to avoid further testing any which do not. A high P value is therefore appropriate: we have a high type I error in order to get a low type II error. If authors wish to do this in published papers, however, they must justify it to the reader, and to the referee.

One of my respondents complained about the use of:

'"Significant" when they mean important.'

This is a difficult one. According to the Shorter Oxford Dictionary, the second meaning of 'significant' is 'important, notable', and has been since 1761. Its statistical meaning relates more to its first definition: 'full of meaning or import'. Thus, if a difference is significant in a sample this difference has meaning, because there is evidence that it exists in the population. I do not think that statisticians can really appropriate 'significant' and deny its other uses, but it's unlikely that I am going to be the statistical referee for your paper, because I do it as rarely as possible. Other statisticians may be more jealous of 'significant' than I and, in the interests of publication, I recommend avoiding its non-statistical applications. The Lancet supports this line, instructing authors to 'Avoid nontechnical uses of technical terms in statistics, such as . . . "significant" . . .'

Another respondent mentioned:

'Direct comparison of p-values.'

I think that what this person had in mind was concluding that one difference is larger or more important than another because it has a smaller P value. This is sometimes done, for example, when a change is tested in two separate groups of subjects and a difference between the P values is interpreted as evidence of a difference between the groups. This is one of my own particular bętes noirs. An example came up in one of my Lancet reviews:

`It is not correct to compare two groups by testing changes in each one separately. Significance does not depend only on magnitude, but on variability and sample size. A two sample t method should be used to compare the log ratios in the two groups.'

One of my respondents made the same point:

'People who carry out controlled clinical trials but do not carry out a controlled analysis. Instead of quoting the estimated treatment effect (active - placebo) with its standard error, they quote the "effect" in the group given active treatment (usually difference from baseline).'

In general we need only note that the P value measures the strength of the evidence that an effect exists in the population, it doesn't convey much about the magnitude of that difference, and a large P value does not, in itself, mean that there is no population difference or that the difference is small.

We must compare effect sizes, not P values. A special case of this was mentioned by another Allstat respondent:

'Sub-group analyses unsupported by interaction tests.'

Sometimes authors will carry out significance tests of the same difference or relationships in different subgroups of their subjects, for example in young and old, male and female. They will then conclude that the difference exists only or mainly in the subgroups where a significant difference was found. As explained above, this conclusion does not follow from the analysis and the correct approach is to test the difference between the magnitudes of the effects in the subgroups (Altman and Matthews 1996; Matthews and Altman 1996a, 1996b; Altman and Bland 2003). This is known as a test of interaction.

Design problems

Referees' criticisms of the study design are the most difficult to deal with. Criticisms of the presentation, analysis, and interpretation of the data can be remedied fairly easily, because all these things can be changed. Once the study has been carried out and the data collected, it cannot be redesigned. It is therefore essential that the design be correct to begin with. Statisticians are forever saying that they should be consulted before the project begins, although, as we are elusive beasts, this is often pretty difficult to achieve.

In my Lancet reports there were only two design issues. The first was a treatment comparison using observational data:

`From a statistical viewpoint, this is pretty awful. I don't think we should have non-randomised clinical trials in the Lancet.'

I think that we have now got past the argument about whether randomised trials are effective or ethical and want to know what the randomised trial evidence for a treatment is. I do not think that randomized trials are the only source of useful information, but authors must be aware of the principles of randomization and have a pretty clear idea of why they are using data from non-randomized subjects and what the limitations of such data are. Two of my Allstat respondents mentioned randomisation. One complained about:

'The adamant refusal of medical investigators to use randomization and random sampling.'

I found this surprising, as in my experience medical investigators are usually very ready to use randomization and there are vast numbers of randomized trials in the literature. However, experience can vary greatly and this informant may have been working in an area of application where trials are few. Sometimes the perspective of others can be startling. In their textbook Using and understanding medical statistics (Matthews and Farewell 1988) the authors wait until chapter 8 before mentioning the Normal distribution, saying that continuous data are rarely encountered in medical research! They devote three chapters to survival analysis. Their experience in cancer research had certainly given them an entirely different perspective to myself, who cut his statistical teeth on peak expiratory flow and forced expiratory volume. When I read that for the first time, my thought was; 'Ever heard of blood pressure?' (Despite this, it's a good book.) However, I entirely agreed with my respondent about random sampling. This is almost unknown in medicine, though usually there are good reasons for this.

Another respondent mentioned:

'Claims that a study is randomised or blinded when in fact allocation has been by hospital number, date of birth, day of week etc, and blinding has been patently superficial and ineffective.'

This is spot on. People who use systematic allocation of this type (hospital number, etc.) sometimes argue that this is random, because the hospital number is not going to be related to the patients' prognosis. But when Bradford Hill first advocated randomisation in clinical trials, it was firstly to avoid such allocation schemes (Chalmers 1999). If clinicians admitting patients to a trial know what treatment the patient will receive, as they will in these systematic systems, this may bias the decision to admit the patient or not. Schulz et al. (1995) have shown that when the admitting clinician is aware of the treatment patients will receive; the treatment effect is larger, on average, than when treatment is concealed. This implies that such open allocation tends to be biased. This might arise, for example, because clinicians might judge a potential trial recruit to be too frail for the trial treatment, but not for the control treatment. They might then decide to recruit the patient to the trial if the patient would receive the control treatment, but not if the patient would receive the trial treatment. Thus a bias in favour of the trial treatment would be built in. Schulz et al. (1995) also showed that trials where the investigators were not blinded to treatment had larger average treatment effects than trials where investigators were blinded. Sometimes blinding is impossible, sometimes it is difficult, but we must always be aware of its importance and the potential for bias when it is not used. I think the referee wants to see that the authors understand this and are suitably cautious in their interpretation as a result. A good point for the discussion.

The other design issue which came up was sample size:

`This is a small trial of two similar regimes. How was the sample size decided? Was there a power calculation? What difference were the authors hoping to detect?'

I have had experience of sample size calculations being removed from papers to shorten them, at the request of the journal. I think we should resist such shortsighted editing, but I think that in this case no sample size calculations, other than feasibility, had been done. I doubted that even had there been the modest treatment effect which they might have hoped for, the chance of getting a significant difference in such a small trial would have been much above 5%. (It is 5% even if there is no difference at all.)

On the subject of sample size, I had no example in my Lancet series, but another thing I would pounce on would be a sample size calculation for a cluster randomized trial which ignored the clustering. I would treat analysis which ignored the clustering in the same way. See other talks: Cluster designs: a personal view, Sample size in guidelines trials, and Cluster randomised trials in the medical literature.

Standard deviation and standard error

Standard deviations and standard errors are the basic currency of statistics, familiar to most researchers, yet they seem to cause a lot of difficulty. One problem is that authors often quote them without specifying what they are quoting. I had two examples in my 15 papers:

`I presume the numbers in brackets are standard deviations. The authors should say so.'

`Are these ± numbers standard deviations, standard errors or confidence intervals?'

One of my respondents also mentioned this:

'± notation without any interpretation of whether it refers to se, sd, or CIs.'

Actually, I find the use of the '±' symbol itself is rather misleading. If we quote 'mean ± SD' as researchers often do, what does this mean? We are not saying that the bservations all lie between mean - SD and mean + SD. In fact, we expect about one third of them to be outside these limits. Similarly, if we quote 'mean ± SD' as researchers often do, what does this mean? We are not saying that the observations all lie between mean - SD and mean + SD. In fact, we expect about one third of them to be outside these limits. Similarly, if we quote 'mean ± SE' we do not actually wish to imply that the population mean lies between mean - SE and mean + SE. This would only be true for 2/3 of samples. I think that standard deviations and standard errors are best placed in parentheses: mean (SD). In one of my papers this ± notation seems to have gone rather haywire:

`There is something wrong with the presentation of X. We have "mean X ... was 51.9 ± 7.9 (range)". Is 7.9 the standard deviation? Have the authors omitted the range by mistake?'

Or did they perhaps mean that the minimum value was 51.9 - 7.9 and the maximum 51.9 + 7.9? This seems most unlikely.

Sometimes the main comparison in a paper is for the same subjects under different conditions, e.g. before and after an intervention. A paired t test might be used. This test uses the mean, standard deviation and standard error of the mean for the differences. Authors often quote the P value from a paired test, but quote the standard deviation or standard error for each condition separately, instead of for differences within the subject. I had a sample of this:

`Most of the standard errors given are irrelevant, as it is the change within subjects which is important, and the standard error of the mean difference is the relevant figure.'

One of my respondents complained about the same thing:

'Confidence intervals (or SE's) on group means, rather than on comparisons.'

If the correct standard deviations and standard errors are given, it is much easier for other workers to incorporate your results in meta-analysis, to compare them with their own data, and so on. Presentation

I had very few comments on specifically on presentation in my Lancet reviews, although my Allstat respondents had quite a lot to say. I made a suggestion that the zero should be included on the y-axis of a graph, and I made this point about a graph:

`I think a scatter plot, showing the actual data, would be much more informative. Are the thin lines standard errors?'

On similar lines, one of my Allstat respondents complained about:

'Dynamite pushers, skyscrapers with TV-aerials'.

What he had in mind, and on which I had been commenting, was a graph like Figure 1:

Bar chart with two bars and two error bars above them. d

Figure 1. Bar graph showing capillary density (per mm2) in the feet of ulcerated patients and a healthy control group (data, but not graph, supplied by Marc Lamah).

You see graphs like this frequently in journals and it may come as a surprise to researchers that many statisticians dislike them intensely. There are several reasons for this. My Allstat respondents complained about:

'Summary graphs with less information than the original data.'

Compare Figure 2:

Two vertical columns of points.  Spread of the columns similar, 'ulcer' group tending to be lower than 'control' group. d

Figure 2. Scatter graph of the capillary density data.

Figure 2 shows the same data as Figure 1 in the form of a scatter diagram or dot plot. This shows not only the relative magnitudes and the variability of the measurement in the two groups, but also the distribution of the measurement. We can add the means and standard deviations to the scatter diagram, as shown in Figure 3:

alt= d

Figure 3. Scatter graph of the capillary density data with mean and standard deviation added.

This now shows all the information in Figure1 and Figure 2. If there are a large number of points, the scatter diagram will become a mass of indistinguishable points. In this case we can use box and whisker plots (see Bland 2000a), as in Figure 4.

Box and whisker plot with two boxes. d

Figure 4. Box and whisker graph of the capillary density data.

These do not give all the information in a scatter diagram, but they do show central tendency, spread and the shape of the distribution. We can see from Figure 4 that the distributions are roughly symmetrical, apart from one rather extreme point, that the control group tend to have higher capillary density that the ulcer group, and that the data are suitable for the t distribution to be applied.

My Allstat respondents had quite a lot to say. A common complaint about graphs such as Figure 1, which I had made in my review, is that authors do not always make clear what the vertical lines represent, standard deviations, standard errors, or confidence intervals, an irritation which I mentioned above concerning '±' notation. A third objection to the bar graph shown in Figure 1 is that it has only four numbers in it, which could be reported much more efficiently in the text. Two of my respondents made similar points:

'Using bar charts to show that the proportion of women in the study was 55% and men 45%, and similar low information ways of using ink and space.' (Two similar replies.)

On the other hand, one respondent complained about:

'Tables of data with (literally) hundreds of figures when the information content is minimal and a graph would be more useful.'

The Lancet instructs its authors to 'Use graphs as an alternative to tables with many entries'. Personally, I am usually inclined to tables rather than graphs. I think that this bias (yes, I have them!) arises because I do not have a strong visual imagination or ability to think pictorially. However, I also think that the argument that other researchers can make use of your findings more easily if they are presented numerically rather than graphically is a forceful one, and this should lead us to choose numbers when in doubt.

I have no problems with the view of my respondents who were irritated by authors:

'Giving far too many decimal places.' (3 replies).

The week before writing this, I reviewed a paper which gave all P values, F statistics, and even degrees of freedom to four decimal places, e.g. 'F=1.9367 with 34.3452 and 45.3298 degrees of freedom, P=0.0189'. This used an approximation to the F distribution which involved changing the degrees of freedom, making them fractional. Now I doubt that the F statistic conveys much useful information anyway, but all those decimal places do not. There is no point in reporting F, t, or chi-squared statistics to more than two decimal places. I do not think that anything would be lost by reducing the decimal places to two here: 'F=1.94 with 34.35 and 45.33 degrees of freedom, P=0.019'. Indeed, I would render the P value to one significant figure: 'P=0.02'. Only the first non-zero number and the number of zeros preceding it are important. The reason for this profligate and unconsidered reporting of many decimal places must be that computer programs deliver them. Programmers try to give the users everything they could possibly want and if the program calculates the F statistic to seven significant figures, why not print them out? But this is no reason for the researcher to burden his readers with them. They often make text and tables much more difficult to read. Correlation coefficients are frequent example. Programs often print them to four decimal places, but is there really any important difference between 'r=0.3421 and 'r=0.3379'? I think that 'r=0.34' would do very nicely for both and make the meaning text and tables easier to grasp.

One respondent complained about something which I also dislike:

'Using multiple crosshatched three-dimensional bars' (2 replies).

I find that three-dimensional effects seldom make a graph clearer. The effect is usually to make it more difficult to read.


Many statistical methods require the data to meet some assumptions, such as that data follow a Normal distribution with uniform variance. Such assumptions are often not checked, particularly for t methods. The statistical referee can often detect skewness from the data and graphs given in the paper (Altman and Bland 1996). One giveaway is a standard deviation which is greater than half the mean, which implies that two standard deviations below the mean would be a negative number. For most measurements negative values are impossible we could not have any observations less than mean minus two standard deviations, and 2.5% of observations from a Normal distribution would be found there. Such data cannot therefore be from a Normal distribution. Another is to give mean or median and quartiles or extreme values. If the mean or median is not close to the centre of the interval determined by the limits, we should suspect that the distribution is skew. Yet another betrayer of non-Normal distributions can arise when the mean and standard deviation or standard error are calculated separately for several different groups, then given in a table or graph. The standard deviation should not be related to the mean. Often we see that groups with large means also have large standard deviations. A scatter diagram of the data, while highly desirable, can also reveal deviations from the assumptions of statistical methods. I three examples of obvious deviations from assumptions in my 15 papers:

`Are the thin lines standard errors? If so, they suggest that the data are not Normal, which casts doubt on the F test.'

`I would be surprised if these measurements followed Normal distributions. Figure 2 suggests that this is not the case, as the distribution of X looks positively skew. The authors should check the distributions of their variables, and use a logarithmic transformation where appropriate.'

`The data are very skewed, positively for X (mean 17.6, range 16.0-21.7) and negatively for Y (mean 8.6, range 4.9-9.4). This is produced by the selection criteria for the trial, which accepts subjects with X > 16.0 and Y < 9.5. No attempt is made to allow for this in the analyses, which assume that data follow Normal distributions.'

To my surprise, only one of my respondents mentioned this:

'Authors who don't attempt to check the normality of their data and use normal theory with clearly non-normal data.'

Incorrect descriptions of statistical methods

The Lancet specifies that authors should: 'Put a general description of methods in the Methods section. When data are summarized in the Results section, specify the statistical methods used to analyze them.' This is good advice. It is certainly annoying when authors do not tell the reader what statistical method is being used and I had an instance in my 15 reviews, in one of which I complained that:

`The statistical test used should be stated.'

My Allstat respondents thought this was an important problem, complaining about:

'Authors who assume that the description of the statistics is so unimportant that they don't actually give any information at all' (5 similar replies).

One had a specific complaint about authors:

'Stating only that "statistical analysis was done using x computer package"'.

Telling us which package was used is important, as they are not all the same and many statistical methods can be implemented in different ways which may give different answers. Indeed, the Lancet asks for it: 'Specify any general-use computer programs used'. But it is not enough to tell us what is being done. In mathematical language, we would say that it is necessary but not sufficient. This reported statistical methods section deserves to become a classic of pointless minimalism:

'The analysis was performed on an IBM486, under MSDOS'

A less frequent, but also irritating, practice is not using the methods stated in the method section of the paper. It is easy to do this, as papers often go through many drafts, with parts being cut out and new one inserted, but it is annoying when an obscure method is references and the referee spends time looking it up only to find that this time had been wasted. I had an example of this in my 15 papers:

`I do not think Hotelling's t test is actually used anywhere.'

An Allstat respondent made the same point:

'Reference in the methods section to analyses undertaken but with no results appearing anywhere in the report.'

This comment from my reviews combined a method reported in the method section which was not used with not saying what done in the analyses which were reported:

`I think that tests other than paired t tests were done. I can't actually find any data suitable for a paired t test. ... the appropriate method would be Fisher's exact test, which gives P=0.2 ... this should be a rank correlation. I get tau=0.37, P=0.08 . . . The appropriate method would be Fisher's exact test, which gives P=0.09.'

I have no idea what they had actually done, but I was pretty confident that whatever it was, was wrong. Sometimes I had to pinch myself to reassure myself that this was not a ghastly nightmare, and that people had really submitted to this stuff to the world's most prestigious medical journal.

Baseline characteristics in randomised trials

Baseline characteristics deserve special mention because two common parastatisical practices relate to them. Baseline characteristics are those which we record after subjects have been recruited to the trial but before treatment begins. There are several good reasons for making and reporting baseline measurements. The first of these is obvious: we want to describe the population which our trial subjects represent. The second is that we want to check and demonstrate that the randomization process has worked. This is not always the case. I was asked to advise on a trial where a programming error had resulted in almost all the older subjects being allocated to one arm of the trial a nd almost all the younger subjects to the other. My advice had to be 'Do it again'. (MacArthur 2001) The third is that we may want to adjust the treatment difference for prognostic variables. If a variable measured at baseline is a strong predictor of the outcome of treatment, adjusting for it statistically may lead to reveal treatment effects which were masked. Altman (1991) gives a good example.

The first common parastatistical mistake is to carry out tests of significance on the baseline variables between the randomized treatment groups. Randomization produces treatment groups which are random samples from the same population. Therefore, any null hypothesis that states that there is no difference between the populations from which the groups come is true. Any significant differences between the treatment groups have arisen by chance; they are type I errors. I had two examples of this in my 15 reviews:

`The tests of significance at baseline should not be done. If the subjects are randomized, they come from the same population and the null hypothesis is true. There is no reason to test it.'

`There is no need to test the difference between the groups before the withdrawal of treatment. Because they are randomised, they are from the same population until treatment is changed, and hence the null hypotheses are true.'

One of my Allstat respondents mentioned this, too, complaining about:

'Significance testing of baseline variables in RCTs.'

The second parastatistical error is that, having tested for differences between baseline characteristics, adjustment of the difference in the outcome measurement between treatments is done for those variables which are significant one the baseline measurements but not for any others. It is not the chance relationship of baseline variables to treatment which is important, but their relationship to the outcome variable. Even when the treatment groups are exactly balanced for the prognostic variable, adjusting for it statistically should remove a lot of variability from the error term and so make confidence intervals narrower and possibly make P values smaller. I had a good example of this approach in one of my reviews:

`The statement that adjustment for baseline characteristics is not needed because baseline differences are not significant is quite wrong. Such adjustments may reduce the variability and so improve the power.'

An Allstat respondent made the same point, complaining about authors:

'Not reporting analyses adjusted for baseline values of prognostic covariates.'

A miscellany

A lot of other issues came up once or twice, either in my own reviews or from my correspondents. I think that this represents the tip of a very large iceberg of possible mistakes on the part of researchers. I present them in the hope that my readers will in future avoid these particular ones at any rate.

An occasional mistake is to include repeated measurements on same subject as if they were different subjects. The data are then analysed using methods which assume that the observations are independent. This can have the effect of making P values too small and confidence intervals too wide. I had a couple of examples in my reviews:

`It is wrong to mix multiple observations from different subjects in this way (Bland and Altman 1994). An appropriate method is described by Bland and Altman (1995).'

`It is not clear why two subjects were measured twice. Inspection of Table 1 suggests that the intention was to measure at 18 hours but that subject 3 was tested additionally at 2 hours and subject 5 at 48 hours. This should be clarified. Repeat observations on the same subject and observations on different subjects cannot be mixed as if they were all independent. I suggest that the first observation on subject 3 and the second on subject 5 should be omitted from the statistical analysis, as they are at very different times.'

The same problem can occur on a larger scale:

`However, they ignore the fact that these 21 groups of subjects are from 9 different trials, and analyse the data as if they are all from the same population.'

Again, this would have the effect of making the P values too small and the confidence intervals too wide. There are well-established mbethods of meta-analysis (see, for example, Bland 2000b) for carrying out the combination of data from different trials and authors should use them.

Significance test methods based on rank order, such as the Mann Whitney and Wilcox on tests and those associated with the Spearman and Kendall rank correlation coefficients, are inappropriate when samples are very small. One cannot have a significant two-sided test at the 5% level when samples are smaller than two groups of four for the Mann Whitney U test or less than six for the Wilcoxon paired test or the rank correlation coefficients. Each possible rank ordering has probability greater than 0.05. Hence rank methods on very small samples are inevitably not significant and there is no point in using them. I made this point in one of my reviews:

`Rank methods are inappropriate for such small samples as they cannot detect any differences, no matter how large the difference is.'

Curiously, I have been asked by publishers to review at least three proposals for introductory statistics text-books (not written by statisticians) which contained the statement that when we have fewer than six observations we should use non-parametric methods, because parametric methods such as t tests are inappropriate, it being impossible to verify the Normal distribution assumptions. The opposite is the case, because parametric methods can produce significant differences for very small samples although rank-based methods cannot. I wish I knew the source of this often-repeated idea. As for checking the Normal assumption, we often have a good idea from other data whether this is reasonable.

Correlation coefficients can cause a problem because there is an assumption that the same is a representative (i.e. random) sample of its population and that both variables are random variables. They should not be used when the values of one variable are set by the experimenter. I had two instances of this in my reviews:

` . . . Correlation is inappropriate when one of the variables is fixed by the investigator (dose and time) . . . One and two sample t methods and regression should be used.'

`The statement that there is no significant correlation between time of measurement and X is meaningless. The times are almost equal except for the duplicate measurements. The ratio is much higher for the early measurement and much lower for the late measurement, suggesting that there is a possibility of a strong relationship with time.'

One my respondents, somewhat enigmatically, cited:

'Spurious use of correlation and regression (oh dear not again!)'

Statisticians mostly have a background in mathematics, as do I, and have been trained for many years to think logically. Indeed, a colleague, Shirley Beresford, once remarked that she thought that the main contribution of statisticians in medical research was not to carry out statistical analyses but 'to inject a bit of logic into the situation'. So imbued with logic are we that we can forget that this is not the only way of thinking and is not the main method of thinking for most people, nor is it always the most useful. Thus to us this one is jaw-dropping:

`The comparisons of X means between the low X and high X groups are not useful. If we divide subjects according X and then compared the mean X between the two groups, of course it will be significant. We could do the same thing with their telephone numbers.'

Of course, the null hypothesis that a group chosen to have X below a cut-off and a group chosen to have X above the cut-off the mean X will be the same is inevitably false. As we know this, there is no point in testing it. I presume the authors simply split the subjects into two groups then tested everything between them. One of my Allstat respondents made a similar point about:

'Dichotomising continuous variables especially if they identify 'responders' and 'non-responders' using these variables.'

Splitting the subjects into two groups using a continuous variable reduces the amount of information which we have. P values may become larger and we may miss important relationships. Some researchers might be tempted to split the sample not at an arbitrary cut-off, such as the overall mean, but to choose a cut-off to minimise a P value and make a relationship significant. This is a real misuse of statistics and will produce misleading results.

The authors of one of the Lancet papers were particularly unlucky (or lucky, depending how you look at it) because they were applying my own work on agreement between methods of measurement and received this comment:

'I suggest replacing the term "95% confidence intervals of agreement" by "95% limits of agreement". The "95% limits of agreement" of Bland and Altman are not a confidence interval, but two point estimates.'

My Allstat respondents came up with a lot more. One mentioned:

'Chi-square test analyses of ordered categorical data.'

What was meant is that we often have categorical data where the categories are ordered in some way, such as physical condition being classified as 'poor', 'fair', 'good' or 'excellent'. The usual chi-squared test for a contingency table ignores this ordering and tests the null hypothesis of no relationship of any sort between the variables. (NEED REAL EXAMPLE HERE.) This is usually a mistake, but an understandable one. Many textbooks use examples with ordered categories to illustrate chi-squared tests.

Another gave the example of

'Rate per 1000 person-years = 3 (95% CI -3 to 9).'

The rate of something per year cannot be negative, so the calculation of the confidence interval has produced an impossible lower limit. This happens because researchers use methods designed for the analysis of large samples or large numbers of events to small samples or small numbers of events. They calculate standard errors and then calculate the confidence interval using the Normal distribution, as the observed value ± 1.96 standard errors. But if the number of events or the sample size is not large enough for this Normal approximation we can get negative lower limits. The same thing can happen with proportions close to the top of their range of possible values, such as sensitivities and specificities, which are sometimes given confidence intervals with upper limits above 100%. There are better approximations and exact methods which can be used in these cases to give confidence intervals which do not include impossible values. Even zero would be an impossible lower limit for the rate in the example, for if in the sample we had observed a case, as we must to get a rate of 3 per 1000 person-years, then the rate in the population cannot be zero. We sometimes see confidence intervals like the one given presented as '3 (95% CI 0 to 9).' This happens because researchers calculate the interval as -3 to 9, recognise that -3 is impossible, and replace it with zero.

My respondents made a couple of general points about the way statistics is carried out in medical research. One complained about:

'Papers where the statistical methods are copied from a previous paper in the field, which was in turn copied from a previous paper, which was in turn . . .'

This undoubtedly happens, and most statisticians have had the experience of researchers who say that a published paper had used a particular method of which the statistician disapproves, and was published, so why shouldn't they? Another respondent complained about:

'Doctors who don't realise that statistics is an advancing science; and the best methods of 20 years ago are not always the best methods of today.'

Well, I think that there are plenty of statisticians in this category, too, and I have no doubt that I am guilty of this from time to time. I do not think we can expect researchers to keep up with what is happening in statistics as well as in their own field. Perhaps, though, we can expect them to embrace a new and better technique when the referee has pointed it out.

One despondent respondent commented:

'There is no hope, at times.'

Not taking us seriously

Some of my respondents complained about authors' attitude to statisticians: These included:

'Papers which show no sign of having had input from a statistician.'

I can sympathise with this, but statisticians can be hard to find for many researchers. The trouble is, you don't know what you don't know, so it hard to spot your own mistakes or to realise that you need help. I think that it should be much easier for researchers to get not just statistical advice but also collaboration. Trying to teach doctors how to analyse their own data is very inefficient. It requires a different way of thinking from medicine, and few people can do both. It is much better to train statisticians to collaborate with them. An additional advantage, unfortunately, is that we do not pay the statistician as much as the doctor, so it makes economic sense too. Another respondent felt that statisticians did not get the prominence they deserved:

'Acknowledgements to a statistician who clearly did all the analysis and should be on the paper.'

Researchers sometimes ask me whether I would like to be acknowledged for my help. I usually paraphrase Oscar Wilde and tell them that there is only one thing worse than being acknowledged, and that is not being acknowledged. I think that the role of the statistician in research is often worthy of authorship, but when I think I am entitled to be an author I am usually welcomed. I think that statisticians have to make clear to researchers who consult them that they have to have something to show for the time they spend in advisory work and that if they make a real contribution, they should be included in the author list. On the other hand, I often refuse authorship because I feel that I have not done enough or could defend the paper.

Two respondents commented on the attitude of authors to statistical referees:

'People who ignore referees comments and send [the] paper to another journal.'

Sometimes this is all an author can do, but I agree that usually authors should take note of what referees say. If, as can happen, the referee has missed the point of the paper entirely, the author should ask why and see how the point can be clarified. Another respondent mentioned:

'The view of many doctors that any comment made by a statistician regarding the quality of the design must by definition be niggling and unimportant.'

I have been accused of being an academic who does not understand the real world of life and death in which doctors operate. This may be true, but so what? I understand something about the world of research and its interpretation. On the whole though, I get on very well with medical profession and have found them warmly welcoming.

The author bites back

Some respondents did not answer my question about what researchers did to annoy referees, but got a few things off their chests about what reviewers did to annoy authors. One complained about:

'Making comments which you know are a matter of opinion and not fact without declaring them as such.'

This is fair enough. If a referee knows that something is only a matter of opinion, they should not condemn others for disagreeing. Another complained about referees:

'Suggesting extensions to analyses which you know will involve far more work than is justified by any likely improvement to the analysis.'

If a referee did really know this then complaints would be justified. Another respondent did not like referees:

'Taking far more time to review a manuscript than is reasonable.'

Mea culpa to that. Refereeing is a difficult task for which one gets little or know reward and which competes for time with the work for which the statistician is paid. Some journal do pay a small fee, but it could not possibly compensate for the time spent in understanding a paper and finding the holes in it. However, I will try to do better.

'Using the anonymity usually afforded to pursue your own interests.'

My own experience as a statistical referee is that I am not remotely interested in the papers which I sent and I am not clear how I could pursue my own interests by impeding their publication. This is more likely to be a complaint about specialist referees who are working in the same area.

'I am giving a pet hate of my own about statistical referees. It is the apparently absolute conviction that their own method of dealing with a data set, whether it be by confidence intervals for differences between groups, their favourite (and usually obscure) measure of agreement, or idiosyncratic ways of normalising data before analysis, is the only right and proper one. In fact, as we all know, a collection of statisticians represents a variance of at least two standard deviations, and they agree to an even lesser extent than psychiatrists. So let's have a bit more humility, please.'

I wondered if the comment about the measure of agreement was a dig at myself. I am quite keen on confidence intervals for differences, too. However, it is certainly true that there is often more than one acceptable way to analyse data. I am irritated by referees who always insist on nonparametric methods because they do not believe that any data follow a Normal distribution, and by those who always insist that nonparametric methods are replaced by parametric ones.

What really upsets me

When I first gave this talk, without the Allstat sample, one of my audience said that he did not think that any of the things I had mentioned really upset me. He thought that what really annoyed me was statistics not being taken seriously by researchers.

I did not think this was the case. I think that what really upset me about this refereeing experience was that there were so many errors in so few papers, and in papers submitted to one of the world's most prestigious medical journals. The journal's own guidelines were ignored. Nothing about most of these papers suggested that the authors had read them.

This suggests a lack of care about research, regarding it as an unimportant activity which does not merit the effort which one hopes these medical researchers put into other aspects of their work. This matters. Incorrect analysis may lead to incorrect conclusions. Incorrect conclusions may lead to incorrect treatments and advice to patients. People can die.

How to avoid upsetting the statistical referee

We can draw a few tentative conclusions from this study. The things which should be avoided above all are:

  1. Read the journal's instructions to authors. If they do not cover statistics, use those of one of the major general medical journals.
  2. Never, ever, conclude that there is no difference or relationship because it is not significant.
  3. Give confidence intervals where you can.
  4. Give exact P values where possible, not P<0.05 or P=NS, though only one significant figure is necessary.
  5. Be clear what your main hypothesis and outcome variable are. Avoid multiple testing.
  6. Get the design right, be clear about blinding and randomisation, do a sample size calculation if you can.
  7. Be clear whether you are quoting standard deviations or standard errors, avoid '±' notation.
  8. Avoid bar charts with error bars.
  9. Check the assumptions of your statistical methods.
  10. Give clear descriptions of your statistical methods.
  11. Decide for which baseline characteristics you should adjust in advance, then do it.

A good aid to writing up clinical trials, and worth reading anyway, is the CONSORT statement (Moher et al., 2001), a template for doing this developed by a group of statisticians and trialists. If you follow this you should sail through the refereeing process.

And finally

I'll finish this talk with three comments from my Lancet reviews:

`The statistics are all wrong but it should be fairly easy to put them right. What a huge number of authors and none of them understand statistics!'

`Why do they do a totally statistical project without a statistician? I suggest they get one!'

And just to show that not all my 15 reviews were negative:

`My comments are very minor, not enough to make me rate any part of the paper as inadequate. I like it.'


I thank Donald Singer for first suggesting the topic, the editors of the Lancet for providing such rich source material, and my Allstat respondents, including Colin Chalmers, Rick Chappell, Tim Cole, Margaret Corbett, Carole Cull, Keith Dear, Michael Dewey, Simon Dunkley, the late Nicola Dollimore, Clarke Harris, Dan Heitjan, Jim Hodges, Alan Kelly, Peter Lewis, Russell Localio, Alison Macfarlane, Sarah MacFarlane, David Mauger, Richard Morris, Ian Plewis, Mike Procter, Paul Seed, Stephen Senn, Jim Slattery, Anthony Staines, Graham Upton, Andy Vail, Ian White, Sheila Williams, Ian Wilson, and a few whose names did not come through with the email.


Altman, D.G. (1991) Practical Statistics for Medical Research Chapman and Hall, London, p. 389-391. Back to text.

Altman DG, Bland JM. (1996) Detecting skewness from summary information. British Medical Journal 313, 1200. Back to text.

Altman DG and Bland JM. (2003) Interaction revisited: the difference between two estimates. 326, 219. Back to text.

Altman DG, Matthews JNS. (1996) Interaction 1: Heterogeneity of effects. British Medical Journal 313, 486. Back to text.

Bland JM, Altman DG. (1994) Correlation, regression and repeated data. British Medical Journal 308, 896. Back to text.

Bland JM, Altman DG. (1995) Calculating correlation coefficients with repeated observations: Part 1, correlation within subjects. British Medical Journal 310, 446. Back to text.

Bland M (2000a) An Introduction to Medical Statistics, 3rd edition Oxford, University Press. Section 4.5 Medians and quantiles. Back to text.

Bland M (2000b) An Introduction to Medical Statistics, 3rd edition Oxford, University Press. Section 17.11 Meta-analysis: data from several studies. Back to text.

Chalmers I. (1999) Why transition from alternation to randomisation in clinical trials was made. British Medical Journal 319, 1372. Back to text.

Gardner, M.J. and Altman, D.G. (1986) Confidence intervals rather than P values: estimation rather than hypothesis testing. British Medical Journal 292, 746-50. Back to text.

MacArthur C, Shennan AH, May A, Whyte J, Hickman N, Cooper G, Bick D, Crewe L, Garston H, Gold L, Lancashire R, Lewis M, Moore P, Wilson M, Bharmal S, Elton C, Halligan A, Hussain W, Patterson M, Squire P, de Swiet M. (2001) Effect of low-dose mobile versus traditional epidural techniques on mode of delivery: a randomised controlled trial. Lancet 358, 19-23. Back to text.

Matthews, D.E. and Farewell, V. (1988) Using and understanding medical statistics, second edition Karger, Basel, Back to text..

Matthews JNS, Altman DG. (1996a) Interaction 2: compare effect sizes not P values. British Medical Journal 313, 808. Back to text.

Matthews JNS, Altman DG. (1996b) Interaction 3: How to examine heterogeneity. British Medical Journal 313, 862. Back to text.

Moher D, Schultz KF, Altman DG. (2001) The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. Lancet 357, 1191-1194. Back to text.

Newnham, J.P., Evans, S.F., Con, A.M., Stanley, F.J., Landau, L.I. (1993) Effects of frequent ultrasound during pregnancy: a randomized controlled trial. Lancet 342, 887-91. Back to text.

Schulz, K.F., Chalmers. I., Hayes, R.J., and Altman, D.G. (1995) Bias due to non-concealment of randomization and non-double-blinding. Journal of the American Medical Association 273, 408-12. Back to text.


From the Lancet's instructions to authors:


Describe statistical methods with enough detail to enable a knowledgeable reader with access to the original data to verify the reported results. When possible, quantify findings and present them with appropriate indicators of measurement error or uncertainty (such as confidence intervals). Avoid relying solely on statistical hypothesis testing, such as the use of P values, which fails to convey important quantitative information. Discuss the eligibility of experimental subjects. Give details about randomization. Describe the methods for and success of any blinding of observations. Report complications of treatment. Give numbers of observations. Report losses to observation (such as dropouts from a clinical trial). References for the design of the study and statistical methods should be to standard works when possible (with pages stated) rather than to papers in which the designs or methods were originally reported. Specify any general-use computer programs used.

Put a general description of methods in the Methods section. When data are summarized in the Results section, specify the statistical methods used to analyze them. Restrict tables and figures to those needed to explain the argument of the paper and to assess its support. Use graphs as an alternative to tables with many entries; do not duplicate data in graphs and tables. Avoid nontechnical uses of technical terms in statistics, such as "random" (which implies a randomizing device), "normal," "significant," "correlations," and "sample." Define statistical terms, abbreviations, and most symbols.

The Lancet's full instructions to authors are well worth reading.

Back to Some full length papers and talks.

Back to Martin Bland's Home Page.

This page is maintained by Martin Bland.
Last updated: 19 August, 2004.