A pilot study on the use of decision theory and value of information analysis as part of the NHS Health Technology Assessment programme

K Claxton,^1,2 L Ginnelly,²* M Sculpher,² Z Philips² and S Palmer²

¹ Department of Economics, University of York, UK
² Centre for Health Economics, University of York, UK

* Corresponding author

Background

This project developed as a result of the investigations of the Research Team at the Centre for Health Economics, University of York, into the methods and application of decision analysis and value of information analysis (DA-VOI) as a means of identifying research priorities, and the interest of the National Coordinating Centre for Health Technology Assessment (NCCHTA) regarding whether these methods might contribute to priority setting in the NHS Health Technology Assessment (HTA) programme. In particular, the potential for DA-VOI to contribute to the process of achieving the greatest return, in terms of outcomes such as health gain, from the resources available to the NHS HTA programme, was a major focus.

Objectives

The specific objectives of the pilot study were to:

demonstrate the benefits of using appropriate decision-analytic methods and value of information analysis
establish the feasibility and resource implications of applying these methods in a timely way, to inform the prioritisation process of the HTA programme
establish the resource implications of adopting these methods more widely within the NHS HTA programme
identify the most appropriate way to extend the use of these methods within the programme’s prioritisation process.

Methods

DA-VOI provides a methodological framework that explicitly considers the uncertainty surrounding the decision of a healthcare system to adopt a health technology. Specifically, using existing evidence, these methods focus on the likelihood of making a wrong decision if the technology is adopted. The value of additional research is based on the extent to which further information will reduce this decision uncertainty. This framework values the additional information that may be generated by further research, in a way that is consistent with the objectives and resource constraints of healthcare provision.

The pilot study relating to the implementation of these methods within the NHS HTA programme was conducted through a series of case studies. It included the application of DA-VOI to three research topics that were considered by the HTA panels in the September 2002 and February 2003 prioritisation rounds: screening in age-related macular degeneration and manual therapy in asthma and in chronic obstructive pulmonary disease. The topic of low-dose antibiotics in children with recurrent urinary tract infections was also considered by the Prioritisation Strategy Group (PSG) in March 2003.

The application of DA-VOI requires three core tasks to be completed: (1) the construction of a decision-analytic model to represent the clinical decision problem being considered; (2) a probabilistic analysis of this model to characterise the current decision uncertainty; and (3) an estimate of the value of additional information through research to reduce decision uncertainty.

A brief and non-technical overview of DA-VOI methods was circulated to the panels and PSG. For each case study the results were presented to the panels and the PSG in the form of brief case-study reports. Feedback on the DA-VOI analysis and its presentation was obtained in the form of completed questionnaires from panel members, and reports from panel senior lecturers and PSG members.

Results

Although none of the research topics identified by NCCHTA met all of the original selection criteria for inclusion as case studies in the pilot, it was possible to construct appropriate decision-analytic models and conduct probabilistic analysis for each topic. In each case, the three core tasks were completed within the time-frame required by the existing HTA research prioritisation process. The brief case-study reports provided a description of the decision problem, a summary of the current evidence base and a characterisation of decision uncertainty in the form of cost-effectiveness acceptability curves. Estimates of value of information for the decision problem were presented for relevant patient groups and clinical settings, as well as the value of information associated with particular model inputs.

The implications for the value of research in each of the areas were presented in general terms. Details were also provided on what the analysis suggested regarding the design of any future research in terms of features such as the relevant patient groups and comparators, and whether experimental design was likely to be required.

Conclusions

The pilot study showed that, even with very short timelines, it is possible to undertake DA-VOI that can feed into the priority-setting process that has developed for the HTA programme.
The use of DA-VOI requires relevant stakeholders to be clear, from an early point in the process, about the nature of the decision problem for which additional research is being considered.
DA-VOI also needs explicitness about which existing data should be used for the first part of the analysis, and how data that exhibit particular weaknesses should be down-weighted in the analysis.
There would be advantages to making the development of the vignette (a summary of the clinical problem and existing evidence) and the use of DA-VOI an integrated process.
It is estimated that each of the pilot studies undertaken required approximately 6 weeks whole-time equivalent researcher input, and this was made up of a mix of experience levels. This research activity needs to be spread out over a period of 10–12 weeks, in part to allow for evidence acquisition.
One approach to the more extensive use of DA-VOI might involve working up a proportion of topics for DA-VOI once they have been identified for a vignette based on existing methods. These analyses would be presented to the panels, along with the vignettes, and they would provide feedback. At the PSG, there would be an analysis that directly addresses the question in the vignette and would include additional analysis to explore any concerns or issues raised by the panel.
Practical considerations about how to implement such methods into a priority-setting system, which has evolved in a particular way, are complex. These include appropriate levels of training for individuals on the relevant panels to achieve the most from DA-VOI, and how analyses of acceptable quality can be assembled in a timely way given limitations of time and skilled resources.
There needs to be some reflection on how the DA-VOI methods handle the heterogeneity and differing levels of quality in the evidence base. Greater use of sensitivity analysis may be a way of handling this problem. Consideration needs to be given to identifying useful scenarios and priorities for sensitivity analysis. This may be an iterative process based on concerns expressed by the panels.
There is a need to identify, and secure access to, relevant clinical experts early in the analysis period when the decision problem is being defined, the structure of the model is being established and relevant data are being identified.
If some degree of implementation of DA-VOI takes place within the HTA programme, careful evaluation and ongoing development will be essential.

Recommendations for research

Methods for efficient literature searching would focus most searching and review attention on those variables to which the model’s results are most sensitive and with the highest expected value of perfect information.
Methods of evidence synthesis (multiple parameter synthesis) to consider the evidence surrounding multiple comparators and networks of evidence.
Ways in which the value of sample information can be used by the NHS HTA programme and other research funders to decide on the most efficient design of new evaluative research.
There is a need for an analytical framework to be developed that can jointly address the question of whether additional resources would better be devoted to additional research or interventions to change clinical practice.

Publication

Claxton K, Ginnelly L, Sculpher M, Philips Z, Palmer S. A pilot study on the use of decision theory and value of information analysis as part of the NHS Health Technology Assessment programme. Health Technol Assess 2004;8(31).

NHS R&D HTA Programme

The research findings from the NHS R&D Health Technology Assessment (HTA) Programme directly influence key decision-making bodies such as the National Institute for Clinical Excellence (NICE) and the National Screening Committee (NSC) who rely on HTA outputs to help raise standards of care. HTA findings also help to improve the quality of the service in the NHS indirectly in that they form a key component of the ‘National Knowledge Service’ that is being developed to improve the evidence of clinical practice throughout the NHS.

The HTA Programme was set up in 1993. Its role is to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and provide care in the NHS. ‘Health technologies’ are broadly defined to include all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care, rather than settings of care.

The HTA programme commissions research only on topics where it has identified key gaps in the evidence needed by the NHS. Suggestions for topics are actively sought from people working in the NHS, the public, consumer groups and professional bodies such as Royal Colleges and NHS Trusts.

Research suggestions are carefully considered by panels of independent experts (including consumers) whose advice results in a ranked list of recommended research priorities. The HTA Programme then commissions the research team best suited to undertake the work, in the manner most appropriate to find the relevant answers. Some projects may take only months, others need several years to answer the research questions adequately. They may involve synthesising existing evidence or designing a trial to produce new evidence where none currently exists.

Additionally, through its Technology Assessment Report (TAR) call-off contract, the HTA Programme is able to commission bespoke reports, principally for NICE, but also for other policy customers, such as a National Clinical Director. TARs bring together evidence on key aspects of the use of specific technologies and usually have to be completed within a limited time period.

Criteria for inclusion in the HTA monograph series

Reports are published in the HTA monograph series if (1) they have resulted from work commissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

The research reported in this monograph was commissioned by the HTA Programmeas project number 02/25/01. As funder, by devising a commissioning brief, the HTA Programme specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the HTA Programme or the Department of Health.

HTA Programme Director: Professor Tom Walley
Series Editors: Dr Peter Davidson, Professor John Gabbay, Dr Chris Hyde, Dr Ruairidh Milne, Dr Rob Riemsma and Dr Ken Stein
Managing Editors: Sally Bailey and Caroline Ciupek

HTA Home Page | Details page for this publication | Publications listing | Publications search