Factors that limit the quality, number and progress of randomised controlled trials

RJ Prescott^1*
CE Counsell²
WJ Gillespie³
AM Grant⁴
IT Russell⁵
S Kiauka¹
IR Colthart¹
S Ross⁴
SM Shepherd⁶
D Russell⁵

¹Medical Statistics Unit, University of Edinburgh, UK
² Department of Clinical Neurosciences, University of Edinburgh, UK
³Department of Orthopaedic Surgery, University of Edinburgh, UK
⁴Health Services Research Unit, University of Aberdeen, UK
⁵Department of Health Sciences, University of York, UK
⁶Department of Public Health Sciences, University of Edinburgh, UK
^*Corresponding author

Background

The randomised controlled trial (RCT) is the most powerful research tool for evaluating health technologies. However, for most therapeutic activities with the NHS, reliable information from RCTs is not available.

Objectives

To assemble and classify a comprehensive bibliography of factors limiting the quality, number and progress of RCTs.
To collate and report the findings, identifying areas where firm conclusions can be drawn, and identifying areas where further research is required.

Methods

A systematic review of the literature was undertaken, covering the period 1986-96. The scope of the review was too broad to be comprehensive in all of the areas covered, rather it attempted to cover the diversity of factors limiting the quality, number and progress of RCTs.

The issues considered were those of design, barriers to participation, conduct and structure, analysis, reporting and costs.

Results and recommendations for practice

Design

Following a systematic review of existing evidence, a well-formulated question should be developed, specifying participants, interventions and outcomes. Wide patient eligibility criteria are generally preferred to give representativeness and good recruitment rates. However, a more homogeneous group may be preferable when evaluating expensive or hazardous interventions. Outcome measures need to be clinically and socially relevant, well-defined, valid, reliable, sensitive to important change and measured at appropriate times. There is evidence that the use of intermediate or surrogate outcomes has been misleading.

The most frequent choice of study design is between a parallel group or a crossover design. Simultaneous investigations of two or more treatments are efficiently approached by using a factorial design. Simple parallel group designs with fixed sample sizes are most common but other designs should be considered.

Protection from selection bias is provided by secure random allocation, using telephone or computer-based randomisation, and by analysis based on the groups as allocated, thus ensuring that groups being compared differ only by chance. Performance bias can be minimised by blinding treatments (when possible) and by employing clearly described treatment policies. Detection bias may be avoided by blind outcome assessment and attrition bias by ensuring follow-up of all patients randomised.

Pre-study sample size calculations should always be made and funding bodies, independent protocol review bodies and journal editors should all demand them. A sensitivity analysis should be considered, with indicative estimates rather than unrealistically precise numbers. Small trials should be reported as hypothesis forming.

Barriers to participation

Barriers to clinician participation include: time constraints, lack of staff and training, concern about the impact on doctor-patient relationships, concern for patients, loss of professional autonomy, difficulty with consent procedures, lack of reward and recognition, and an insufficiently interesting question.

Barriers to patient participation include: additional demands of the trial, patient preferences, concern caused by uncertainty and concerns about information and consent.

To overcome barriers to clinician recruitment, a trial should address an important research question and the protocol and data collection should be as straightforward as possible, with demands on clinicians and patients kept to a minimum. Dedicated research staff may be required to support clinical staff and patients. The recruitment aspects of an RCT should be carefully planned and piloted.

Conduct and structure

Many trials fail to start, mainly because of lack of funding or logistical problems. Of those that start, half have recruitment difficulties, leading to abandonment or reduced size and, hence, loss of statistical power. Recruitment problems may be reduced by piloting, using multiple recruitment strategies, making contingency plans in case recruitment is slow, and using recruitment coordinators. None of these approaches has been rigorously evaluated.

Inadequate compliance with the study protocol can lead to false-negative or false-positive results. Some assessment of compliance (clinician and participant) should be made but may be difficult to measure.

Quality control is important but too much may make RCTs prohibitively expensive and hinder recruitment. Trials need good organisational and administrative bases but there is little research evaluating the optimal structure. The precise roles of steering committees and data monitoring committees have been poorly evaluated. There is concern about bias in the design, conduct, analysis and reporting of commercially sponsored trials, and independent monitoring should be considered.

Analysis

Intention-to-treat analysis is the method of choice to provide an unbiased estimate of treatment effects. In studies where the aims are more explanatory than pragmatic, consideration should be given to reporting analysis by treatment received as well as intention-to-treat.

Study protocols should identify a predetermined primary outcome supplemented by secondary outcomes and a clear statistical plan. Any subgroup analyses that are proposed as hypothesis testing should be specified in the protocol and the study must be of sufficient size to detect such an interaction. All other subgroup analyses should be considered as hypothesis-generating.

Reporting

The introduction of the Consolidation of Standards for Reporting Trials (CONSORT) guidelines should improve reporting of RCTs. Conclusions should be supported by the data presented. About 10% of trials remain un-published while many others are only published in conference proceedings, particularly if they are small and show non-significant treatment effects: prospective registration of all RCTs is recommended. Multiple publication of a study is also a problem for studies showing significant results.

Costs

Economic evaluations are reported in few RCTs, possibly because of difficulties in conducting such evaluations and the lack of generalisability from one healthcare context to another. Some components of an economic analysis are subject to uncertainty; statistical tests and confidence intervals should, therefore, be used.

There has been little research into trial costs but costs of caring for patients in RCTs may be perceived as an unaffordable new service, delaying or preventing recruitment at some participating centres.

Conclusions

The evidence available to guide many aspects of the design, conduct and analysis of RCTs is not always being applied.

Recommendations for research

Further research is required, particularly in relation to:

problems being experienced and solutions employed in current RCTs
the optimum structure, staffing and organisation for the conduct of large and small trials
the factors which influence the participation of clinicians and patients in trials.

Publication

Prescott RJ, Counsell CE, Gillespie WJ, Grant AM, Russell IT, Kiauka S, et al. Factors that limit the quality, number and progress of randomised controlled trials. Health Technol Assessment 1999; 3(20).

NHS R&D HTA Programme

The overall aim of the NHS R&D Health Technology Assessment (HTA) programme is to ensure that high quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and work in the NHS. Research is undertaken in those areas where the evidence will lead to the greatest benefits to patients, either through improved patient outcomes or the most efficient use of NHS resources.

The Standing Group on Health Technology advises on national priorities for health technology assessment. Six advisory panels assist the Standing Group in identifying and prioritising projects. These priorities are then considered by the HTA Commissioning Board supported by the National Coordinating Centre for HTA.

This report is one of a series covering acute care, diagnostics and imaging, methodology, pharmaceuticals, population screening, and primary and community care. It was identified as a priority by the Methodology Group and funded as project number 93/43/02. The views expressed in this publication are those of the authors and not necessarily those of the Standing Group, the Commissioning Board, the Panel members or the Department of Health. The editors wish to emphasize that funding and publication of this research by the NHS should not be taken as implicit support for the recommendations for policy contained herein. In particular, policy options in the area of screening will be considered by the National Screening Committee. This Committee, chaired by the Chief Medical Officer, will take into account the views expressed here, further available evidence and other relevant considerations. Reviews in Health Technology Assessment are termed 'systematic' when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

Series Editors: Andrew Stevens, Ruairidh Milne, Ken Stein
Assistant Editor: Melanie Corris

The editors have tried to ensure the accuracy of this report but cannot accept responsibility for any errors or omissions. They would like to thank the referees for their constructive comments on the draft document.

HTA Home Page | Details page for this publication | Publications listing | Publications search