Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system

P Royle*
N Waugh

Department of Public Health, University of Aberdeen, UK

*Corresponding author

Background

In the UK, one part of the remit of the National Institute for Clinical Excellence (NICE) is to carry out a programme of technology appraisals. These are done to a fairly tight timetable in order not to delay the guidance on new technologies. Each appraisal is underpinned by a Technology Assessment Report (TAR) commissioned from a group of academic units.

As the TAR process is relatively new, and is still evolving, the methods used for its literature searching have been largely based on the well-established and documented methods used for Cochrane reviews. These involve comprehensive searching of a variety of sources to protect against bias, but can add substantially to the time and costs of carrying out a review.

However, resource constraints require that TARs are produced as efficiently as possible, and to a tight timetable, which means that not all of the Cochrane methods can be applied, or are appropriate. In addition, it is not known whether the marginal benefits of exhaustive searching justify the costs. The challenge for those undertaking TARs is to know how best to adapt and optimise, and extend when necessary, the Cochrane-based search strategies, so that searching can be done both rapidly and systematically.

Objective

To contribute to making searching for TARs more cost-effective by suggesting an optimum literature retrieval strategy, based on empirical data obtained from a sample of recent TARs, which balances comprehensiveness and efficiency.

Methods

A sample of 20 recent TARs was studied. All sources used to search for clinical and cost-effectiveness studies were recorded. In addition, all studies that were included in the clinical and cost-effectiveness sections of the TARs were identified, and their characteristics recorded, including author, journal, year, study design, study size and quality score. Each was also classified by publication type, and then checked to see whether it was indexed in the following databases: MEDLINE, EMBASE, and then either the Cochrane Controlled Trials Register (CCTR) for clinical effectiveness studies or the NHS Economic Evaluation Database (NHS EED) for the cost-effectiveness studies. Any study not found in at least one of these databases was checked to see whether it was indexed in the Science Citation Index (SCI) and BIOSIS, and the American Society of Clinical Oncology (ASCO) Online if a cancer review. Any studies still not found were investigated further to see whether they were in a number of additional databases.

Results

Sources searched

The median number of sources searched per TAR was 20, and the range was from 13 to 33 sources. Six sources (CCTR, DARE, EMBASE, MEDLINE, NHS EED and sponsor/industry submissions to NICE) were used in all reviews.

Clinical effectiveness studies

There were 424 studies in total. The publication types were: published 80%, meeting abstracts 11.3% and unpublished 8.7%. Eighty per cent of reviews included at least one abstract or unpublished study (60% included at least one abstract and 50% included at least one unpublished study). The median number of studies included per TAR was 19.5 (range 2–41). The median number of participants included per TAR was 2787 (range 69–97,570). Evidence from non-randomised controlled trial (RCT) studies was used in 45% of TARs. The proportion of studies classified either as published in full or as abstracts, and found indexed in the following databases, was: MEDLINE 82.7%, EMBASE 78.6% and CCTR 50.1%. The cumulative percentage of studies found after searching these three databases was 87.3%. Adding SCI, BIOSIS and ASCO Online increased this to 98.2%. Eighty-seven per cent of studies were indexed in both MEDLINE and EMBASE.

Cost-effectiveness studies

The 130 studies were classified as: published 73.1%, unpublished 23.8%, abstracts 1.5% and grey literature 1.5%. The median number of studies used was 4.0. The percentage of studies classified as either published in full or as abstracts, and found indexed in the following databases, was: MEDLINE 86.6%, EMBASE 86.6% and NHS EED 40.2%. The cumulative percentage of these studies found indexed after searching the three databases was 94.8%. Adding SCI and ASCO Online increased this to 97.9%.

Studies used in the economic modelling

The 121 articles were classified as: published 50.4%, abstracts 5.0%, reference sources 17.4%, unpublished 17.4% and grey literature 9.8%. The median number of studies used for the 14 TARs that included an economic model was 9.0 per TAR.

Search terms for identifying non-RCTs

A sensitive search filter, constructed for MEDLINE and using the search terms from the bibliographic records in the included studies, retrieved only 85% of the known sample. Therefore, it is recommended that when searching for non-RCT studies a search is done for the intervention alone, and records are then scanned manually for those that look relevant.

Conclusions

Searching additional databases beyond the Cochrane Library (which includes CCTR, NHS EED and the HTA database), MEDLINE, EMBASE and SCI, plus BIOSIS limited to meeting abstracts only, is seldom effective in retrieving additional studies for inclusion in the clinical and cost-effectiveness sections of TARs (apart from reviews of cancer therapies, where a search of the ASCO database is recommended). A more selective approach to database searching would suffice in most cases and would save resources, thereby making the TAR process more efficient. However, searching non-database sources (including submissions from manufacturers, recent meeting abstracts, contact with experts and checking reference lists) does appear to be a productive way of identifying further studies.

Publication

Royle P, Waugh N. Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. Health Technol Assess 2003;7(34).

NHS R&D HTA Programme

The NHS R&D Health Technology Assessment (HTA) Programme was set up in 1993 to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and provide care in the NHS.

The research reported in this monograph was commissioned by the HTA Programme and funded as project number 02/30/01. Technology assessment reports are completed in a limited time to inform decisions in key areas by bringing together evidence on the use of the technology concerned.

The views expressed in this publication are those of the authors and not necessarily those of the HTA Programme or the Department of Health. The editors wish to emphasise that funding and publication of this research by the NHS should not be taken as implicit support for any recommendations made by the authors.

Criteria for inclusion in the HTA monograph series

Reports are published in the HTA monograph series if (1) they have resulted from work commissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

HTA Programme Director: Professor Kent Woods

Series Editors: Professor Andrew Stevens, Dr Ken Stein, Professor John Gabbay, Dr Ruairidh Milne, Dr Chris Hyde and Dr Rob Riemsma

Managing Editors: Sally Bailey and Sarah Llewellyn Lloyd

The editors and publisher have tried to ensure the accuracy of this report but do not accept liability for damages or losses arising from material published in this report.

HTA Home Page | Details page for this publication | Publications listing | Publications search