St. George's Hospital Medical School

Statistics Guide for Research Grant Applicants

Authors (in alphabetical order) JM Bland, BK Butland, JL Peacock, J Poloniecki, F Reid, P Sedgwick.

Edited by BK Butland, prepared for the World Wide Web by JM Bland.

We welcome comments. Please include the words "comments on guide" in the subject line.
Email comments on content to Barbara Butland.
Email comments on Web implementation (broken links, etc.) to Martin Bland.

Department of Public Health Sciences
St George's Hospital Medical School
Cranmer Terrace
London SW17 0RE

This handbook was funded by the South East Regional Office (SERO) Research and Knowledge Management Directorate and is targeted at those applying for research funding, from any source.

Brief Table of Contents

  • Title Page and top
  • Detailed Table of Contents
  • Preface
  • A Describing the study design
  • B Clinical Trials
  • C Observational Studies
  • D Sample Size Calculation
  • E Describing the statistical methods
  • F General
  • References
  • Appendix
  • 1. Check list
  • 2. Directory of randomisation software and services
  • Detailed Table of Contents

  • Preface
  • Background
  • Content
  • Using the handbook
  • Statistical review
  • Development
  • A Describing the study design
    Introduction
    A-1 Type of study
    A-1.1 Type of study: Observational or experimental
    A-1.2 Combinations and sequences of studies
    A-1.3 Cohort studies
    A-1.4 Case-control studies
    A-1.5 Cross-sectional studies
    A-1.5a Prevalence studies
    A-1.5b The estimation of sensitivity and specificity
    A-1.5c When to calculate sensitivity and specificity
    A-1.5d Cross-sectional ecological studies
    A-1.5e Studies of measurement validity, reliability and agreement
    A-1.6 Confounding
    A-1.6a Confounding or interaction
    A-1.7 Experiments and trials
    A-1.8 Randomised controlled clinical trials
    A-1.9 Pilot and Exploratory Studies
    A-2 Follow-up
    A-3 Study subjects
    A-4 Types of variables
    A-4.1 Scales of measurement
    A-4.2 Types of data
    A-4.3 Methods of data collection
    A-4.4 Validity and reliability
    A-4.4a Validity
    A-4.4b Repeatability (test-re-test reliability)
    A-4.4c Inter-rater reliability (inter-rater agreement)
  • B Clinical trials
    B-1 Describing the patient group / eligibility criteria
    B-2 Describing the intervention (s) / treatment (s)
    B-3 Choice of control treatment / need for control group
    B-4 Blindness
    B-4.1 Double blind and single blind designs B-4.2 Placebos B-5 Randomisation
    B-5.1 What is randomisation?
    B-5.2 When might we use randomisation?
    B-5.3 Why randomise?
    B-5.4 What is not randomisation?
    B-5.5 How do we randomise?
    B-5.6 Randomisation in blocks
    B-5.7 Randomisation in strata
    B-5.8 Minimisation
    B-5.9 Clusters
    B-5.10 Trial designs
    B-5.10a Parallel groups
    B-5.10b Crossover design
    B-5.10c Within group comparisons
    B-5.10d Sequential design
    B-5.10e Factorial design
    B-6 Outcome variables
    B-7 Data monitoring
    B-7.1 Data monitoring committee
    B-7.2 When should a trial be stopped early?
    B-8 Informed consent
    B-8.1 Consent
    B-8.2 Emergencies
    B-8.3 Children
    B-8.4 Mentally incompetent subjects
    B-8.5 Cluster randomised designs
    B-8.6 Random consent designs
    B-9 Protocol violation and non-compliance
    B-10 Achieving the sample size
    B-11 After the trial is over
  • C Observational studies
    C-1 Case-control studies
    C-1.1 Choice of control group in case-control studies
    C-1.2 Matching in case-control studies
    C-2 Assessment bias
    C-3 Recall bias
    C-4 Sample survey: selecting a representative sample
    C-5 Generalisability and extrapolation of results
    C-6 Maximising response rates to questionnaire surveys
  • D Sample size calculation
    D-1 When should sample size calculations be provided?
    D-2 Why is it important to consider sample size?
    D-3 Information required to calculate a sample size
    D-4 Explanation of statistical terms
    D-4.1 Null and alternative hypothesis
    D-4.2 Probability value (p-value)
    D-4.3 Significance level
    D-4.4 Power
    D-4.5 Effect size of clinical importance
    D-4.6 One-sided and two-sided tests of significance
    D-5 Which variables should be included in the sample size calculation?
    D-6 Allowing for response rates and other losses to the sample
    D-7 Consistency with study aims and statistical analysis
    D-8 Three specific examples of sample size calculations & statements
    D-8.1 Estimating a single proportion
    D-8.2 Comparing two proportions
    D-8.3 Comparing two means
    D-9 Sample size statements likely to be rejected
  • E Describing the statistical methods
    E-1 Introduction E-1.1 Terminology
    E-1.2 Level of detail required
    E-2 Is the proposed method appropriate for the data?
    E-2.1 'Ordinal' scores
    E-3 Paired and unpaired comparison
    E-4 Assumptions
    E-4.1 Transformations
    E-5 Adjustment for confounding
    E-6 Hierarchical or multilevel data
    E-6.1 Analysing hierarchical data
    E-7 Multiple testing
    E-7.1 Multiple testing: when does it arise?
    E-7.2 Multiple testing: why is it a problem?
    E-7.3 The Bonferroni correction
    E-7.4 How to deal with multiple testing
    E-7.4a More than one outcome measurement in a clinical trial
    E-7.4b More than one predictor measurement in an observational study
    E-7.4c Measurements repeated over time (serial measurements)
    E-7.4d Comparisons of more than two groups
    E-7.4e Testing the study hypothesis within subgroups
    E-7.4f Repeatedly testing the difference in a study as more patients are recruited
    E-8 Change over time (regression towards the mean)
    E-9 Intention to treat in clinical trials
    E-10 Cluster randomised trials
    E-11 Collapsing variables
    E-12 Estimation and confidence intervals
    E-12.1 Proportions close to 1 or zero
  • F General
    F-1 Statistical expertise (statistical analysis)
    F-2 Statistical software
    F-3 Ethics
    F-3.1 The misuse of statistics
    F-3.2 Critical appraisal
    F-3.3 Studies involving human subjects
    F-4 Other research issues
    F-4.1 Research governance
    F-4.2 Data protection
  • References
  • Appendix
  • 1. Check list
  • 2. Directory of randomisation software and services
  • Preface

    Background

    The aim of this handbook is to help applicants to appreciate some of the statistical pitfalls that await them when constructing a grant proposal. It was written by the following six statisticians based at St George's Hospital Medical School:

    Martin Bland PhD - Professor of Medical Statistics
    Barbara Butland MSc - Lecturer in Medical Statistics
    Janet Peacock PhD - Senior Lecturer in Medical Statistics
    Jan Poloniecki DPhil - Senior Lecturer in Medical Statistics
    Fiona Reid MSc - Lecturer in Medical Statistics
    Philip Sedgwick PhD - Lecturer in Medical Statistics

    All six authors routinely reviewed grant proposals for The South East Research and Development Project Grant Scheme. This was a responsive funding scheme for the South East Regional Office which ran for a number of years until it ended in October 2001. The scheme was responsible for spending of approximately 1.2 million each year on new and ongoing projects, all based on researchers' own ideas. Its primary objective was the production of new, high quality knowledge. It was open to anyone whose work was relevant to the UK National Health Service (NHS). Up to 150,000 pounds was available for each project, although the majority were smaller. The criteria stipulated that applications were to be relevant to the NHS, to follow a clear, well defined protocol, would withstand external peer review and that the findings would be generalisable to others in the NHS. Although academic advice was available to applicants through Research and Development Support Units (RDSU) in the Region and detailed guidance notes accompanied the application form, a number of common statistical problems consistently emerged. It is hoped that in summarising these statistical weak points the handbook will be of use to those applying for research funding from any source and in the long term reduce the number of proposals that are rejected by grant giving bodies purely on statistical grounds. The handbook was commissioned and initially funded by the South East Regional Office (SERO) Research and Knowledge Management Office Directorate who provided comments and encouragement throughout the writing process. In this respect the authors would particularly like to extend their thanks to John Newton and Lesley Elliott.

    Back to top

    Content

    The process of constructing the handbook began with a pilot exercise. All six authors retrieved their reviews for the last year to two years noting down recurring comments. Based on the results of the pilot work a list of topics was drawn up to form the main skeleton of the handbook. A few additional topics have since been added at the suggestion of Colin Cryer (South East Institute of Public Health) a fellow statistical reviewer for The Project Grant Scheme and Lyn Fletcher (Statistician, Oxford Research and Development Support Unit).

    Back to top

    Using the handbook

    The handbook is not designed to teach statistics but to provide extra information to those who already have a basic statistical knowledge. For example it assumes some understanding of confidence intervals and significance testing but not statistical power or sample size calculation. To help the beginner there are references to standard Medical Statistics Textbooks. However, the handbook should not be viewed as an alternative to discussing your proposal with a statistician prior to submission. Such a discussion is strongly recommended. Rather, it is hoped that the handbook will make grant applicants more aware of the right questions to ask and the right information to take along to a statistical consultation and in addition, help them to understand any advice given. [Please note that although the authors would welcome any comments on the content of the guide, particularly correction of errors, they cannot give advice on projects]

    It is not envisaged that the handbook should be read from cover to cover but rather that the contents list or the checklist (Appendix 2) should be used to help the applicant navigate through the book ignoring sections that have no bearing on their own research. To facilitate this type of use the text is organised into many short reasonably self-contained paragraphs, each with its own index code e.g. A-1.1. It is hoped that these codes will be useful to reviewers, consulting statisticians and researchers alike. Each paragraph may contain links to other related paragraphs as well as to useful references in the literature and on the Web. The handbook is best suited to interactive use on the Web although it may also be used effectively in printed form.

    Back to top

    Statistical review

    Another aim of the handbook is to try and clarify the sort of checklist that a statistician might use in the process of reviewing a grant proposal. Most statisticians will not rigidly follow any such list but the sort of things that they will be trying to extract from the text of any grant proposal are as follows (this list is not exhaustive):

    a) the basic study design(s) to see whether the applicant needs to have included information on randomisation, confounding, hierarchical data etc and whether the design is appropriate to the study aims.

    b) the type of data the study will generate as without this information the statistician cannot assess whether the applicant's sample size calculation and proposed methods of statistical analysis are appropriate.

    c) the number of subjects that will be asked to take part in the study and the number it is anticipated will be recruited as the figure from the sample size calculation should match the latter and not the former.

    d) the total number of outcome variables that the applicant plans to measure as this will highlight potential problems in terms of multiple testing.

    d) whether the sample size calculation and the proposed statistical analysis are based on the same statistical tests. The applicant should therefore mention when reporting the sample size calculation(s) the test(s) on which it is based.

    e) whether the proposed statistical analysis includes the calculation of confidence intervals as well as significance testing.

    f) whether the applicants have the required statistical expertise for their proposed statistical analysis.

    g) sufficient information to replicate and so check the applicant's sample size calculation.

    Back to top

    Development

    The handbook should be considered as a work in progress and the authors would welcome any comments on the content of the guide, particularly correction of errors.

    Back to top


    Public Health Sciences Home Page

    St. George's Hospital Medical School Home Page

    This page is maintained by Martin Bland
    Last updated: [September 11, 2002]

    df=sample.dat&dd=B

    Copyright St. George's Hospital Medical School

    Back to top