Statistics Guide for Research Grant Applicants
Authors (in alphabetical order)
JM Bland, BK Butland, JL Peacock, J Poloniecki, F Reid, P Sedgwick.
Edited by BK Butland, prepared for the World Wide Web by JM Bland.
We welcome comments. Please include the words "comments on guide" in the
subject line.
Email comments on content to Barbara Butland.
Email comments on Web implementation (broken links, etc.) to Martin Bland.
Division of Community Health Sciences
St George's University of London
Cranmer Terrace
London SW17 0RE
This handbook was funded by the NHS South East Regional Office (SERO) Research and
Knowledge Management Directorate and is targeted at those applying for research
funding, from any source.
Detailed Table of Contents

Preface

A Describing the study design
Introduction
A1 Type of study
A1.1 Type of study: Observational or experimental
A1.2 Combinations and sequences of studies
A1.3 Cohort studies
A1.4 Casecontrol studies
A1.5 Crosssectional studies
A1.5a Prevalence studies
A1.5b The estimation of sensitivity and specificity
A1.5c When to calculate sensitivity and specificity
A1.5d Crosssectional ecological studies
A1.5e Studies of measurement validity, reliability and agreement
A1.6 Confounding
A1.6a Confounding or interaction
A1.7 Experiments and trials
A1.8 Randomised controlled clinical trials
A1.9 Pilot and Exploratory Studies
A2 Followup
A3 Study subjects
A4 Types of variables
A4.1 Scales of measurement
A4.2 Types of data
A4.3 Methods of data collection
A4.4 Validity and reliability
A4.4a Validity
A4.4b Repeatability (testretest reliability)
A4.4c Interrater reliability (interrater agreement)

B Clinical trials
B1 Describing the patient group / eligibility criteria
B2 Describing the intervention (s) / treatment (s)
B3 Choice of control treatment / need for control group
B4 Blindness
B4.1 Double blind and single blind designs
B4.2 Placebos
B5 Randomisation
B5.1 What is randomisation?
B5.2 When might we use randomisation?
B5.3 Why randomise?
B5.4 What is not randomisation?
B5.5 How do we randomise?
B5.6 Randomisation in blocks
B5.7 Randomisation in strata
B5.8 Minimisation
B5.9 Clusters
B5.10 Trial designs
B5.10a Parallel groups
B5.10b Crossover design
B5.10c Within group comparisons
B5.10d Sequential design
B5.10e Factorial design
B6 Outcome variables
B7 Data monitoring
B7.1 Data monitoring committee
B7.2 When should a trial be stopped early?
B8 Informed consent
B8.1 Consent
B8.2 Emergencies
B8.3 Children
B8.4 Mentally incompetent subjects
B8.5 Cluster randomised designs
B8.6 Random consent designs
B9 Protocol violation and noncompliance
B10 Achieving the sample size
B11 After the trial is over

C Observational studies
C1 Casecontrol studies
C1.1 Choice of control group in casecontrol studies
C1.2 Matching in casecontrol studies
C2 Assessment bias
C3 Recall bias
C4 Sample survey: selecting a representative sample
C5 Generalisability and extrapolation of results
C6 Maximising response rates to questionnaire surveys

D Sample size calculation
D1 When should sample size calculations be provided?
D2 Why is it important to consider sample size?
D3 Information required to calculate a sample size
D4 Explanation of statistical terms
D4.1 Null and alternative hypothesis
D4.2 Probability value (pvalue)
D4.3 Significance level
D4.4 Power
D4.5 Effect size of clinical importance
D4.6 Onesided and twosided tests of significance
D5 Which variables should be included in the sample size calculation?
D6 Allowing for response rates and other losses to the sample
D7 Consistency with study aims and statistical analysis
D8 Three specific examples of sample size calculations & statements
D8.1 Estimating a single proportion
D8.2 Comparing two proportions
D8.3 Comparing two means
D9 Sample size statements likely to be rejected

E Describing the statistical methods
E1 Introduction
E1.1 Terminology
E1.2 Level of detail required
E2 Is the proposed method appropriate for the data?
E3 Paired and unpaired comparison
E4 Assumptions
E5 Adjustment for confounding
E6 Hierarchical or multilevel data
E6.1 Analysing hierarchical data
E7 Multiple testing
E7.1 Multiple testing: when does it arise?
E7.2 Multiple testing: why is it a problem?
E7.3 The Bonferroni correction
E7.4 How to deal with multiple testing
E7.4a More than one outcome measurement in a clinical trial
E7.4b More than one predictor measurement in an observational study
E7.4c Measurements repeated over time (serial measurements)
E7.4d Comparisons of more than two groups
E7.4e Testing the study hypothesis within subgroups
E7.4f Repeatedly testing the difference in a study as more patients are
recruited
E8 Change over time (regression towards the mean)
E9 Intention to treat in clinical trials
E10 Cluster randomised trials
E11 Collapsing variables
E12 Estimation and confidence intervals
E12.1 Proportions close to 1 or zero

F General
F1 Statistical expertise (statistical analysis)
F2 Statistical software
F3 Ethics
F3.1 The misuse of statistics
F3.2 Critical appraisal
F3.3 Studies involving human subjects
F4 Other research issues
F4.1 Research governance
F4.2 Data protection

References
 Appendix
Preface
Background
The aim of this handbook is to help applicants to appreciate some of the
statistical pitfalls that await them when constructing a grant proposal. It was
written by the following six statisticians based at St George's Hospital
Medical School:
Martin Bland PhD  Professor of Medical Statistics
Barbara Butland MSc  Lecturer in Medical Statistics
Janet Peacock PhD  Senior Lecturer in Medical Statistics
Jan Poloniecki DPhil  Senior Lecturer in Medical Statistics
Fiona Reid MSc  Lecturer in Medical Statistics
Philip Sedgwick PhD  Lecturer in Medical Statistics
(Martin Bland is now Prof. of Health Statistics, University of York, and
Janet Peacock is Prof. of Medical Statistics, University of Southampton.)
All six authors routinely reviewed grant proposals for The South East Research
and Development Project Grant Scheme. This was a responsive funding scheme for
the South East Regional Office which ran for a number of years until it ended
in October 2001. The scheme was responsible for spending of approximately £1.2
million each year on new and ongoing projects, all based on researchers' own
ideas. Its primary objective was the production of new, high quality knowledge.
It was open to anyone whose work was relevant to the UK National Health Service
(NHS). Up to 150,000 pounds was available for each project, although the
majority were smaller. The criteria stipulated that applications were to be
relevant to the NHS, to follow a clear, well defined protocol, would withstand
external peer review and that the findings would be generalisable to others in
the NHS. Although academic advice was available to applicants through Research
and Development Support Units (RDSU) in the Region and detailed guidance notes
accompanied the application form, a number of common statistical problems
consistently emerged. It is hoped that in summarising these statistical weak
points the handbook will be of use to those applying for research funding from
any source and in the long term reduce the number of proposals that are
rejected by grant giving bodies purely on statistical grounds. The handbook was
commissioned and initially funded by the South East Regional Office (SERO)
Research and Knowledge Management Office Directorate who provided comments and
encouragement throughout the writing process. In this respect the authors would
particularly like to extend their thanks to John Newton and Lesley Elliott.
Back to top
The process of constructing the handbook began with a pilot exercise. All six
authors retrieved their reviews for the last year to two years noting down
recurring comments. Based on the results of the pilot work a list of topics was
drawn up to form the main skeleton of the handbook. A few additional topics
have since been added at the suggestion of Colin Cryer (South East Institute of
Public Health) a fellow statistical reviewer for The Project Grant Scheme and
Lyn Fletcher (Statistician, Oxford Research and Development Support Unit).
Back to top
The handbook is not designed to teach statistics but to provide extra
information to those who already have a basic statistical knowledge. For
example it assumes some understanding of confidence intervals and significance
testing but not statistical power or sample size calculation. To help the
beginner there are references to standard Medical Statistics Textbooks.
However, the handbook should not be viewed as an alternative to discussing your
proposal with a statistician prior to submission. Such a discussion is strongly
recommended. Rather, it is hoped that the handbook will make grant applicants
more aware of the right questions to ask and the right information to take
along to a statistical consultation and in addition, help them to understand
any advice given. [Please note that although the authors would welcome any
comments on the content of the guide, particularly correction of errors, they
cannot give advice on projects]
It is not envisaged that the handbook should be read from cover to cover but
rather that the contents list or the checklist (Appendix 2) should be used to
help the applicant navigate through the book ignoring sections that have no
bearing on their own research. To facilitate this type of use the text is
organised into many short reasonably selfcontained paragraphs, each with its
own index code e.g. A1.1. It is hoped that
these codes will be useful to reviewers, consulting statisticians and
researchers alike. Each paragraph may contain links to other related paragraphs
as well as to useful references in the literature and on the Web. The handbook
is best suited to interactive use on the Web although it may also be used
effectively in printed form.
Back to top
Another aim of the handbook is to try and clarify the sort of checklist that a
statistician might use in the process of reviewing a grant proposal. Most
statisticians will not rigidly follow any such list but the sort of things that
they will be trying to extract from the text of any grant proposal are as
follows (this list is not exhaustive):
a) the basic study design(s) to see whether the applicant needs to have
included information on randomisation, confounding, hierarchical data etc and
whether the design is appropriate to the study aims.
b) the type of data the study will generate as without this information the
statistician cannot assess whether the applicant's sample size calculation and
proposed methods of statistical analysis are appropriate.
c) the number of subjects that will be asked to take part in the study and the
number it is anticipated will be recruited as the figure from the sample size
calculation should match the latter and not the former.
d) the total number of outcome variables that the applicant plans to measure as
this will highlight potential problems in terms of multiple testing.
d) whether the sample size calculation and the proposed statistical analysis
are based on the same statistical tests. The applicant should therefore mention
when reporting the sample size calculation(s) the test(s) on which it is based.
e) whether the proposed statistical analysis includes the calculation of
confidence intervals as well as significance testing.
f) whether the applicants have the required statistical expertise for their
proposed statistical analysis.
g) sufficient information to replicate and so check the applicant's sample size
calculation.
Back to top
The handbook should be considered as a work in progress and the authors would
welcome any comments on the content of the guide, particularly correction of
errors.
Back to top
To St. George's Community Health Sciences
Home Page
To St. George's University of London Home Page
Back to Martin Bland's home page.
This page is maintained by Martin Bland.
Last updated: 10 September, 2009.
Copyright St. George's University of London
Back to top