Issues in data monitoring and interim analysis of trials

AM Grant,^1* DG Altman,² AB Babiker,³ MK Campbell,¹ FJ Clemens,⁴ JH Darbyshire,³ DR Elbourne,⁴ SK McLeer,¹ MKB Parmar,³ SJ Pocock,⁴ DJ Spiegelhalter,⁵ MR Sydes,³ AF Walker,¹ SA Wallace¹ and the DAMOCLES study group

¹ Health Services Research Unit, University of Aberdeen, UK
² Cancer Research UK Medical Statistics Group, Centre for Statistics in Medicine, Institute of Health Sciences, Oxford, UK
³ MRC Clinical Trials Unit, London, UK
⁴ Medical Statistics Unit, London School of Hygiene and Tropical Medicine, UK
⁵ MRC Biostatistics Unit, Institute of Public Health, University of Cambridge, UK

* Corresponding author

Objectives

To address issues about data monitoring committees (DMCs) for randomised controlled trials (RCTs): why and when they are needed, their roles and responsibilities, their structure and organisation, what information is required and who owns it, and decision-making and reporting arrangements.

Methods

The study included systematic literature reviews of DMCs and small group processes in decision-making; sample surveys of: reports of RCTs, recently completed and ongoing RCTs and policies of major organisations involved in RCTs; case studies of four DMCs; and interviews with experienced DMC members. All focused on 23 prestated questions.

Results

Although still a minority, RCTs increasingly have DMCs. There is wide agreement that nearly all trials need some form of data monitoring. Criteria suggested for RCTs not needing an independent DMC are: where it is not possible for a DMC to make a contribution, where any observed differences would not prompt any protocol change (such as early stopping), and where there is no reason why a DMC’s decisions would differ from those after internal monitoring.

A range of roles has been suggested for DMCs. Central is monitoring accumulating evidence related to benefit and toxicity; variation in emphasis has been reflected in the plethora of names. DMCs for trials performed for regulatory purposes should be aware of any special requirements and regulatory consequences.

Advantages were identified for both larger and smaller DMCs. There is general agreement that a DMC should be independent (no commercial, clinical or intellectual competing interests) and multidisciplinary (at least one statistician and one clinician). Consumer and ethicist membership is controversial. The chair is recognised as being particularly influential, and likely to be most effective if he or she is experienced, understands both statistical and clinical issues, and is facilitating in style and impartial. There is no evidence available to judge suggested approaches to training.

The review suggested that costs should be covered, but other rewards must be so minimal as to not affect decision-making.

It is usual to have a minimum frequency of DMC meetings, with the committee able to meet at shorter notice. There is evidence that face-to-face meetings are preferable, especially for the first meeting or when difficult decisions are predicted; teleconferencing can be used when the discussion is expected to be straightforward or when there are practical difficulties convening the committee. It is common to have open sessions (where general issues, such as recruitment, are discussed with investigators) and a closed session (where confidential information, such as interim analyses, is discussed by the DMC supported by the analysis statistician).

The general view is that a report to a DMC should cover benefits and risks in a balanced way, summarised in an accessible style, avoiding excessive detail, and as current as possible. Disadvantages of blinded analyses seem to outweigh advantages. Information about comparable studies should be included, although interaction with the DMCs of similar ongoing trials is controversial.

A range of formal statistical approaches can be used. However, this is only one of a number of considerations that a DMC should take into account. DMCs usually reach decisions by consensus, but other approaches are sometimes used. The general, but not unanimous, view is that DMCs should be advisory rather than executive on the basis that it is the trial organisers who are ultimately responsible for the conduct of the trial.

Conclusions

The conclusions of the study are summarised below.

Some form of data monitoring should be considered for all RCTs, with reasons given where there is no DMC or when any member is not independent.

An early DMC meeting is helpful. Roles, responsibilities and planned operations can be agreed with investigators and sponsors/funders. A template for a DMC charter is suggested. Competing interests should be declared.

DMC size (commonly three to eight people) is chosen to optimise performance. Members are usually independent and drawn from appropriate backgrounds, and some, particularly the chair, are experienced. Hitherto, members have received little training.

A minimum frequency of meetings is usually agreed, with flexibility for more if needed. Meetings are best held face-to-face, if practicable. There are advantages of having both open and closed sessions. Often, the trial’s statistician conducts the confidential analyses and attends the closed sessions (but not as a member).

The DMC should understand and agree the statistical approach (and guidelines) chosen, with both the DMC statistician and analysis statistician competent to apply the method.

A DMC’s primary purpose is to ensure that continuing a trial according to its protocol is ethical, taking account of both individual and collective ethics. A broader remit in respect of wider ethical issues is controversial; arguably, these are primarily the responsibility of research ethics committees, trial steering committees and investigators.

The DMC should know the range of recommendations or decisions open to it, in advance. A record should be kept describing the key issues discussed and the rationale for decisions taken.

Errors are likely to be reduced if a DMC makes a thorough review of the evidence and has a clear understanding of how it should function, there is active participation by all members, differences are resolved through discussion and there is systematic consideration of the various decision options.

DMCs should be encouraged to comment on draft final trial reports. These should include information about the data monitoring process and detail the DMC membership.

It is recommended that groups responsible for data monitoring be given the standard name ‘Data Monitoring Committee’ (DMC).

Recommendations for research

Areas that warrant further research include:

widening DMC membership beyond clinicians, trialists and statisticians (e.g. to include consumer representatives or ethicists)
initiatives to train DMC members
methods of DMC decision-making, such as voting and formal decision-making tools
‘open’ data monitoring
DMCs covering a portfolio of trials rather than single trials
DMC size and membership, incorporating issues of group dynamics
empirical study of the workings of DMCs and their decision-making
which trials should or should not have a DMC.

Publication

Grant AM, Altman DG, Babiker A, Campbell MK, Clemens F, Darbyshire J, et al. Issues in data monitoring and interim analysis of trials. Health Technol Assess 2005;9(7).

NHS R&D HTA Programme

The research findings from the NHS R&D Health Technology Assessment (HTA) Programme directly influence key decision-making bodies such as the National Institute for Clinical Excellence (NICE) and the National Screening Committee (NSC) who rely on HTA outputs to help raise standards of care. HTA findings also help to improve the quality of the service in the NHS indirectly in that they form a key component of the ‘National Knowledge Service’ that is being developed to improve the evidence of clinical practice throughout the NHS.

The HTA Programme was set up in 1993. Its role is to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and provide care in the NHS. ‘Health technologies’ are broadly defined to include all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care, rather than settings of care.

The HTA programme commissions research only on topics where it has identified key gaps in the evidence needed by the NHS. Suggestions for topics are actively sought from people working in the NHS, the public, consumer groups and professional bodies such as Royal Colleges and NHS Trusts.

Research suggestions are carefully considered by panels of independent experts (including consumers) whose advice results in a ranked list of recommended research priorities. The HTA Programme then commissions the research team best suited to undertake the work, in the manner most appropriate to find the relevant answers. Some projects may take only months, others need several years to answer the research questions adequately. They may involve synthesising existing evidence or designing a trial to produce new evidence where none currently exists.

Additionally, through its Technology Assessment Report (TAR) call-off contract, the HTA Programme is able to commission bespoke reports, principally for NICE, but also for other policy customers, such as a National Clinical Director. TARs bring together evidence on key aspects of the use of specific technologies and usually have to be completed within a limited time period.

Criteria for inclusion in the HTA monograph series

Reports are published in the HTA monograph series if (1) they have resulted from work commissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

The research reported in this monograph was commissioned by the HTA Programmeas project number 98/25/06. As funder, by devising a commissioning brief, the HTA Programme specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the HTA Programme or the Department of Health.

Editor-in-Chief: Professor Tom Walley
Series Editors: Dr Peter Davidson, Professor John Gabbay, Dr Chris Hyde, Dr Ruairidh Milne, Dr Rob Riemsma and Dr Ken Stein
Managing Editors: Sally Bailey and Caroline Ciupek

HTA Home Page | Details page for this publication | Publications listing | Publications search