Identification and assessment of ongoing trials in health technology assessment reviews

FJ Song,^1* A Fry-Smith,² C Davenport,² S Bayliss,² Y Adi,² JS Wilson² and C Hyde²

¹ Institute of Health, University of East Anglia, Norwich, UK
² Department of Public Health and Epidemiology, University of Birmingham, UK

* Corresponding author

Background and objectives

Clinical and policy decisions on healthcare interventions have to be made according to the best currently available evidence. However, the evidence base evolves over time. Knowledge about the existence of ongoing trials and considering their possible impact on research evidence will help decision-makers to understand how confident or tentative their decisions must be. The awareness and assessment of ongoing research may result in more appropriate decisions about whether and when a completed health technology assessment review (HTAR) should be updated. Any recommendations for further trials should also consider trials in progress. This research aims to assess the importance of ongoing trials in HTARs for the National Institute for Clinical Excellence and to provide practical recommendations for identifying ongoing trials and assessing their possible impact.

Methods

Ongoing trials (or trials in progress) were defined as any trials that have started but where the results are not yet available or only interim results are available for HTARs. This methodological review included: (1) an assessment of ongoing trials in HTARs completed by the end of August 2002, (2) a survey and assessment of trial registers and other sources of ongoing trials and (3) a summary and assessment of available methods for assessing the possible impacts of ongoing trials.

Ongoing trials in the completed HTARs

The identification of ongoing trials was found to be a common phenomenon in reviews of health technology assessment. Twenty-three of the 32 HTARs identified one or more ongoing trials. This phenomenon was not clearly associated with any HTAR characteristics, such as disease or technology categories, explicitness of search strategies, convincingness of HTAR conclusions and number of studies included. In eight of the 23 HTARs with ongoing trials, the information on identified ongoing trials was not considered in the evidence synthesis and research recommendations. Of the remaining 15 HTARs with ongoing trials, 12 attempted to consider the impact of ongoing trials on conclusions, eight on research recommendations and only three HTARs with ongoing trials incorporated information on ongoing trials in the results synthesis. All but one HTAR that considered the potential impact of ongoing trials adopted a narrative approach.

Sources of and searching for ongoing trials

Trial registers and grey literature are important sources of information on ongoing trials. There are a large number of trial registers (international or national, general or subject-specific). The assessment of six commonly used trial registers suggested that most registers provided sufficient information for reviewers to decide the relevance of identified ongoing trials. However, it was sometimes extremely difficult to know whether ongoing trials identified from different sources (registers) were the same trials or belonged to the same multicentre trials. The ISRCTN (the International Standard Randomised Controlled Trial Number) is the most reliable system but it has not been widely adopted. All 32 HTARs explicitly or implicitly searched for unpublished studies, and/or ongoing trials and/or grey literature and trial registers. The efforts made to search for unpublished trials or grey literature may result in the identification of ongoing trials. Case studies indicated that a search of additional sources may identify additional ongoing trials.

Methods for assessing the impact of ongoing trials

The qualitative assessment of ongoing trials compared major features of completed and ongoing trials, providing information about the possible impact of ongoing trials in terms of relevance, validity, reliability and generalisability. All quantitative methods that may be used to assess the impact of ongoing trials require subjective judgement about levels of Type I and II error, minimal clinically worthwhile benefit, and presumed prior distribution of the parameter. The fail-safe N method and Bayesian data monitoring method do not directly use information on ongoing trials, but focus on the assessment of the conclusiveness of existing evidence. The number of patients in ongoing trials may be useful for estimating optimal or cumulative information size for cumulative meta-analysis-related methods (sequential monitoring boundaries and stochastic curtailment method). The most useful method may be the Bayesian predictive probability, which estimates predictive probabilities for any possible values of treatment effect. A case study indicated that the appropriate use of quantitative methods would strengthen findings from narrative assessment of possible impact of ongoing trials.

Conclusions

Identification of ongoing trials is common in HTARs. Searching for ongoing trials in effectiveness reviews should be more thorough and explicit. Conversely, primary researchers, in particular those working within multicentre trials, should label ongoing trials more clearly, preferably by ISRCTN. Qualitative assessment of identified ongoing trials is crucial and informative. Available quantitative methods could be used to strengthen findings from narrative assessment, although further research and more empirical examples are required. Information from ongoing trials may contribute to syntheses of results, conclusions and recommendations for future research.

Recommendations for future research

The following areas are suggested for further research.

Identification and assessment of ongoing trials in other systematic reviews of effectiveness of healthcare interventions (for example, Cochrane Systematic Reviews) should be evaluated.
Existing and new qualitative and quantitative methods for incorporating information on ongoing trials need to be tested and compared in further effectiveness reviews and/or computer simulation studies.
The validity of estimated impacts of ongoing trials could be evaluated by comparing estimated impacts with the actual results of ongoing trials. This could be done prospectively with long-term follow-up of selected HTARs. A retrospective study would also be possible by examining the evolution of trial evidence for selected topics.
Further research is required to incorporate findings from the assessment of ongoing trials into decision models. For example, posterior predictive distribution may be useful for dealing with uncertainty problems in cost-effectiveness modelling.

Publication

Song FJ, Fry-Smith A, Davenport C, Bayliss S, Adi Y, Wilson JS, et al. Identification and assessment of ongoing trials in health technology assessment reviews. Health Technol Assess 2004;8(44).

NHS R&D HTA Programme

The research findings from the NHS R&D Health Technology Assessment (HTA) Programme directly influence key decision-making bodies such as the National Institute for Clinical Excellence (NICE) and the National Screening Committee (NSC) who rely on HTA outputs to help raise standards of care. HTA findings also help to improve the quality of the service in the NHS indirectly in that they form a key component of the ‘National Knowledge Service’ that is being developed to improve the evidence of clinical practice throughout the NHS.

The HTA Programme was set up in 1993. Its role is to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and provide care in the NHS. ‘Health technologies’ are broadly defined to include all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care, rather than settings of care.

The HTA programme commissions research only on topics where it has identified key gaps in the evidence needed by the NHS. Suggestions for topics are actively sought from people working in the NHS, the public, consumer groups and professional bodies such as Royal Colleges and NHS Trusts.

Research suggestions are carefully considered by panels of independent experts (including consumers) whose advice results in a ranked list of recommended research priorities. The HTA Programme then commissions the research team best suited to undertake the work, in the manner most appropriate to find the relevant answers. Some projects may take only months, others need several years to answer the research questions adequately. They may involve synthesising existing evidence or designing a trial to produce new evidence where none currently exists.

Additionally, through its Technology Assessment Report (TAR) call-off contract, the HTA Programme is able to commission bespoke reports, principally for NICE, but also for other policy customers, such as a National Clinical Director. TARs bring together evidence on key aspects of the use of specific technologies and usually have to be completed within a limited time period.

Criteria for inclusion in the HTA monograph series

Reports are published in the HTA monograph series if (1) they have resulted from work commissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

The research reported in this monograph was commissioned by the HTA Programmeas project number 02/28/01. As funder, by devising a commissioning brief, the HTA Programme specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the HTA Programme or the Department of Health.

Editor-in-Chief: Professor Tom Walley

Series Editors: Dr Peter Davidson, Professor John Gabbay, Dr Chris Hyde, Dr Ruairidh Milne, Dr Rob Riemsma and Dr Ken Stein

Managing Editors: Sally Bailey and Caroline Ciupek

HTA Home Page | Details page for this publication | Publications listing | Publications search