Health Technology Assessment 1999; Vol. 3: No. 20
monograph in Adobe Acrobat format (664 kbytes)
Factors that limit the quality, number and progress of randomised controlled trials
1Medical Statistics Unit, University of Edinburgh, UK
The randomised controlled trial (RCT) is the most powerful research tool for evaluating health technologies. However, for most therapeutic activities with the NHS, reliable information from RCTs is not available.
A systematic review of the literature was undertaken, covering the period 1986-96. The scope of the review was too broad to be comprehensive in all of the areas covered, rather it attempted to cover the diversity of factors limiting the quality, number and progress of RCTs.
The issues considered were those of design, barriers to participation, conduct and structure, analysis, reporting and costs.
Results and recommendations for practice
Following a systematic review of existing evidence, a well-formulated question should be developed, specifying participants, interventions and outcomes. Wide patient eligibility criteria are generally preferred to give representativeness and good recruitment rates. However, a more homogeneous group may be preferable when evaluating expensive or hazardous interventions. Outcome measures need to be clinically and socially relevant, well-defined, valid, reliable, sensitive to important change and measured at appropriate times. There is evidence that the use of intermediate or surrogate outcomes has been misleading.
The most frequent choice of study design is between a parallel group or a crossover design. Simultaneous investigations of two or more treatments are efficiently approached by using a factorial design. Simple parallel group designs with fixed sample sizes are most common but other designs should be considered.
Protection from selection bias is provided by secure random allocation, using telephone or computer-based randomisation, and by analysis based on the groups as allocated, thus ensuring that groups being compared differ only by chance. Performance bias can be minimised by blinding treatments (when possible) and by employing clearly described treatment policies. Detection bias may be avoided by blind outcome assessment and attrition bias by ensuring follow-up of all patients randomised.
Pre-study sample size calculations should always be made and funding bodies, independent protocol review bodies and journal editors should all demand them. A sensitivity analysis should be considered, with indicative estimates rather than unrealistically precise numbers. Small trials should be reported as hypothesis forming.
Barriers to participation
Barriers to clinician participation include: time constraints, lack of staff and training, concern about the impact on doctor-patient relationships, concern for patients, loss of professional autonomy, difficulty with consent procedures, lack of reward and recognition, and an insufficiently interesting question.
Barriers to patient participation include: additional demands of the trial, patient preferences, concern caused by uncertainty and concerns about information and consent.
To overcome barriers to clinician recruitment, a trial should address an important research question and the protocol and data collection should be as straightforward as possible, with demands on clinicians and patients kept to a minimum. Dedicated research staff may be required to support clinical staff and patients. The recruitment aspects of an RCT should be carefully planned and piloted.
Conduct and structure
Many trials fail to start, mainly because of lack of funding or logistical problems. Of those that start, half have recruitment difficulties, leading to abandonment or reduced size and, hence, loss of statistical power. Recruitment problems may be reduced by piloting, using multiple recruitment strategies, making contingency plans in case recruitment is slow, and using recruitment coordinators. None of these approaches has been rigorously evaluated.
Inadequate compliance with the study protocol can lead to false-negative or false-positive results. Some assessment of compliance (clinician and participant) should be made but may be difficult to measure.
Quality control is important but too much may make RCTs prohibitively expensive and hinder recruitment. Trials need good organisational and administrative bases but there is little research evaluating the optimal structure. The precise roles of steering committees and data monitoring committees have been poorly evaluated. There is concern about bias in the design, conduct, analysis and reporting of commercially sponsored trials, and independent monitoring should be considered.
Intention-to-treat analysis is the method of choice to provide an unbiased estimate of treatment effects. In studies where the aims are more explanatory than pragmatic, consideration should be given to reporting analysis by treatment received as well as intention-to-treat.
Study protocols should identify a predetermined primary outcome supplemented by secondary outcomes and a clear statistical plan. Any subgroup analyses that are proposed as hypothesis testing should be specified in the protocol and the study must be of sufficient size to detect such an interaction. All other subgroup analyses should be considered as hypothesis-generating.
The introduction of the Consolidation of Standards for Reporting Trials (CONSORT) guidelines should improve reporting of RCTs. Conclusions should be supported by the data presented. About 10% of trials remain un-published while many others are only published in conference proceedings, particularly if they are small and show non-significant treatment effects: prospective registration of all RCTs is recommended. Multiple publication of a study is also a problem for studies showing significant results.
Economic evaluations are reported in few RCTs, possibly because of difficulties in conducting such evaluations and the lack of generalisability from one healthcare context to another. Some components of an economic analysis are subject to uncertainty; statistical tests and confidence intervals should, therefore, be used.
There has been little research into trial costs but costs of caring for patients in RCTs may be perceived as an unaffordable new service, delaying or preventing recruitment at some participating centres.
The evidence available to guide many aspects of the design, conduct and analysis of RCTs is not always being applied.
Recommendations for research
Further research is required, particularly in relation to:
Prescott RJ, Counsell CE, Gillespie WJ, Grant AM, Russell IT, Kiauka S, et al. Factors that limit the quality, number and progress of randomised controlled trials. Health Technol Assessment 1999; 3(20).
NHS R&D HTA Programme
The overall aim of the NHS R&D Health Technology Assessment (HTA) programme is to ensure that high quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and work in the NHS. Research is undertaken in those areas where the evidence will lead to the greatest benefits to patients, either through improved patient outcomes or the most efficient use of NHS resources.
The Standing Group on Health Technology advises on national priorities for health technology assessment. Six advisory panels assist the Standing Group in identifying and prioritising projects. These priorities are then considered by the HTA Commissioning Board supported by the National Coordinating Centre for HTA.
This report is one of a series covering acute care, diagnostics and imaging,
methodology, pharmaceuticals, population screening, and primary and community care. It was
identified as a priority by the Methodology Group and funded as project number 93/43/02.
The views expressed in this publication are those of the authors and not necessarily those
of the Standing Group, the Commissioning Board, the Panel members or the Department of
Health. The editors wish to emphasize that funding and publication of this research by the
NHS should not be taken as implicit support for the recommendations for policy contained
herein. In particular, policy options in the area of screening will be considered by the
National Screening Committee. This Committee, chaired by the Chief Medical Officer, will
take into account the views expressed here, further available evidence and other relevant
considerations. Reviews in Health Technology Assessment are termed 'systematic'
when the account of the search, appraisal and synthesis methods (to minimise biases and
random errors) would, in theory, permit the replication of the review by others.
The editors have tried to ensure the accuracy of this report but cannot accept responsibility for any errors or omissions. They would like to thank the referees for their constructive comments on the draft document.
©1999 Crown Copyright