Impact of computer-aided detection prompts on the sensitivity and specificity of screening mammography

P Taylor,¹* J Champness,² R Given-Wilson,³ K Johnston⁴ and H Potts⁵

¹ Centre for Health Informatics and Multiprofessional Education (CHIME), Royal Free and University College Medical School, London, UK
² Cancer Services Collaborative, St George’s Hospital, London, UK
³ Radiology Department, St George’s Hospital, London, UK
⁴ Scottish Executive, Edinburgh, UK
⁵ Adamson Centre, St Thomas’ Hospital, London, UK

* Corresponding author

Background

This report describes two studies carried out in order to assess the potential role of computer aids in the UK NHS Breast Screening Programme (NHSBSP).

Objectives

The objective was to determine the value of computer-aided detection (CAD) for breast cancer screening. The impact of the R2 ImageChecker^® on the sensitivity and specificity of radiologists and film-reading radiographers was assessed in two experiments, referred to here as study 1 and study 2, and the resulting data were used in an economic evaluation.

Methods

Design

Two sets of mammograms with known outcomes were used. Participants in both studies read the films with and without the benefit of the computer aid. In both studies, the order of reading sessions was randomised separately for each reader. The first set of 180 films, used in study 1, included 20 false-negative interval cancers and 40 screen-detected cancers. The second set of 120 films, used in study 2, was designed to be favourable to CAD: all 44 cancer cases had previously been missed by a film reader and cancers prompted by CAD were preferentially included.

Setting

The studies were conducted at five screening centres: South-West London, Norfolk and Norwich, Luton and Dunstable, Worthing, and Bristol. Study 1 was conducted between January 2001 and July 2002, and study 2 between September 2002 and April 2003.

Participants

Thirty radiologists, five breast clinicians and 15 radiographers participated.

Interventions

All cases in the trial were digitised and analysed using the R2 ImageChecker version 2.2. Participants all received training on the use of CAD. In the intervention condition participants interpreted cases with a prompt sheet on which regions of potential abnormality were indicated.

Main outcome measures

The sensitivity and specificity of participants were measured in both intervention and control conditions.

Results

No significant difference was found for readers’ sensitivity or specificity between the prompted and unprompted conditions in study 1 [95% confidence index (CI) for sensitivity with and without CAD is 0.76 to 0.80, for specificity it is 0.81 to 0.86 without CAD and 0.81 to 0.87 with CAD]. No statistically significant difference was found between the sensitivity and specificity of the different groups of film reader (95% CI for unprompted sensitivity of radiologists was 0.75 to 0.81, for radiographers it was 0.71 to 0.81, prompted sensitivity was 0.76 to 0.81 for radiologists and 0.69 to 0.79 for radiographers). Thirty-five readers participated in study 2. Sensitivity was improved in the prompted condition (0.81 from 0.78) but the difference was slightly below the threshold for statistical significance (95% CI for the difference –0.003 to 0.064). Specificity also improved (0.87 from 0.86); again, the difference was not significant at 0.05 (95% CI –0.003 to 0.034). A cost-effectiveness analysis was performed based on data from studies 1 and 2. The analysis showed that computer prompting is cost-increasing.

Conclusions and recommendations for research

No significant improvement in film readers’ sensitivity or specificity or gain in cost-effectiveness was established in either study. This may be due to the system’s low specificity, its relatively poor sensitivity for subtle cancers and the fact the prompts cannot serve as aids to decision-making. It may be that readers would be better able to make use of the prompts if they had longer to become accustomed to working with them. Prompts may have an impact in routine use that is not detectable in an experimental setting.

Although the case for CAD as an element of the NHSBSP is not made here, further research is required. Evaluations of new CAD tools in routine use are underway and their results should be given careful attention.

There should be a clearer and speedier route to commissioning evaluations of rapidly changing technologies.

Publication

Taylor P, Champness J, Given-Wilson R, Johnston K, Potts H. Impact of computer-aided detection prompts on the sensitivity and specificity of screening mammography. Health Technol Assess 2005;9(6).

NHS R&D HTA Programme

The research findings from the NHS R&D Health Technology Assessment (HTA) Programme directly influence key decision-making bodies such as the National Institute for Clinical Excellence (NICE) and the National Screening Committee (NSC) who rely on HTA outputs to help raise standards of care. HTA findings also help to improve the quality of the service in the NHS indirectly in that they form a key component of the ‘National Knowledge Service’ that is being developed to improve the evidence of clinical practice throughout the NHS.

The HTA Programme was set up in 1993. Its role is to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and provide care in the NHS. ‘Health technologies’ are broadly defined to include all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care, rather than settings of care.

The HTA programme commissions research only on topics where it has identified key gaps in the evidence needed by the NHS. Suggestions for topics are actively sought from people working in the NHS, the public, consumer groups and professional bodies such as Royal Colleges and NHS Trusts.

Research suggestions are carefully considered by panels of independent experts (including consumers) whose advice results in a ranked list of recommended research priorities. The HTA Programme then commissions the research team best suited to undertake the work, in the manner most appropriate to find the relevant answers. Some projects may take only months, others need several years to answer the research questions adequately. They may involve synthesising existing evidence or designing a trial to produce new evidence where none currently exists.

Additionally, through its Technology Assessment Report (TAR) call-off contract, the HTA Programme is able to commission bespoke reports, principally for NICE, but also for other policy customers, such as a National Clinical Director. TARs bring together evidence on key aspects of the use of specific technologies and usually have to be completed within a limited time period.

Criteria for inclusion in the HTA monograph series

Reports are published in the HTA monograph series if (1) they have resulted from work commissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

The research reported in this monograph was commissioned by the HTA Programmeas project number 98/16/04. As funder, by devising a commissioning brief, the HTA Programme specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the HTA Programme or the Department of Health.

Editor-in-Chief: Professor Tom Walley
Series Editors: Dr Peter Davidson, Professor John Gabbay, Dr Chris Hyde, Dr Ruairidh Milne, Dr Rob Riemsma and Dr Ken Stein
Managing Editors: Sally Bailey and Caroline Ciupek

HTA Home Page | Details page for this publication | Publications listing | Publications search