Handling uncertainty when performing economic evaluation of healthcare interventions

AH Briggs
AM Gray

Health Economics Research Centre, Institute of Health Sciences, University of Oxford, UK

Aims

To perform a structured review of the way in which uncertainty has been handled in economic evaluation.
To assemble data on the actual distributional form and variance of healthcare costs, and to devise guidelines to improve current practice. In particular, the focus was on the handling of cost and cost-effectiveness data.

Methods

The structured review was conducted at a number of different levels, reflecting the detail of the review process. At a general level, a search of the literature was undertaken to identify published economic evaluation studies that reported results in terms of cost per life-year or cost per quality-adjusted life-year values. This form of study was chosen as it is the results of these studies that are commonly grouped together and reported in cost-effectiveness league tables. Articles meeting the search criteria were reviewed using a review proforma designed to collect summary information on each study. These results were then entered as key words into a database, to allow interrogation and cross-referencing of the database by category.

This overall data set was then employed to focus in on two specific areas of interest, using subsets of articles to perform more detailed reviews:

All studies reporting UK results were identified from the wider group of articles. These studies were reviewed in detail, and information on the baseline cost-effectiveness results, the methods underlying those results, the range of results representing uncertainty and the number of previously published results quoted in comparison were entered into a relational database. By matching results by the methods employed using a retrospective application of a methodological 'reference case', a subset of results with improved comparability was identified, and a rank ordering of these results was then attempted. Where a range of values accompanied the baseline results, the implications of this uncertainty for the rank ordering was also examined.
All studies which reported patient level cost data were identified from the overall database and reviewed in detail with respect to how they had reported the distribution and variance of healthcare costs. In addition, five available data sets of patient level cost data were examined in order to show how the healthcare costs in those data were distributed and to elucidate issues surrounding the analysis and presentation of healthcare cost differences.

Economic analyses are not simply concerned with costs but also with effects, with the cost-effectiveness ratio being the outcome of interest in most economic evaluations. Unfortunately, ratio statistics pose particular problems for standard statistical methods. In this report, a review of a number of proposed methods for estimating confidence limits for cost-effectiveness ratios when patient level data are available for both cost and effectiveness is presented.

Results

A total of 492 articles were found to match the search criteria, and were fully reviewed and entered into the database. Analysis of this database in terms of the method employed by analysts to handle uncertainty shows that the vast majority of studies use one-way sensitivity analysis methods only. Of some concern is that 17% of studies did not attempt any analysis to examine uncertainty, although there is weak evidence to show that this situation is improving.

Of these 492 studies, 60 reported results for the UK. From these UK studies, 548 baseline cost-effectiveness results were extracted relating to 106 methodological scenarios. Application of a retrospective 'reference case' gave a single methodological scenario for each article with 333 associated baseline results. These results were converted to a common cost base year, and rank ordered to give a comprehensive 'league table' of UK results. Of the 333 results, 61 had an associated full range of values to represent uncertainty. Alternative rankings based on the high or low values from this range showed that there could be considerable disruption to the rank order based on the baseline point estimates only.

The review of patient level cost data showed that 53 of the 492 studies in the database had patient level cost data and that just 15 of these had reported some measure of cost variance. Only four studies had calculated 95% confidence intervals for cost. The review of five available cost data sets showed that the cost data were not normally distributed, and in two cases showed substantial skewness.

A number of methods for estimating confidence intervals for cost-effectiveness ratios have appeared in the recent literature. Examination of their statistical properties and evidence from recent Monte Carlo simulation studies suggests that many of these methods may not perform well in some circumstances. The parametric method based on Fieller's theorem and the non-parametric approach of bootstrapping produced consistently the best results, and are the preferred methods for estimating confidence intervals for cost-effectiveness ratios. However, the use of cost-effectiveness acceptability curves may provide more useful information to decision makers than standard confidence intervals.

Conclusions

General recommendations

Potential guidelines arising from this review are:

analysts should aim to present results using a methodological reference case in order to increase the comparability of results between studies
analysts should be aware of the potential for the incremphasizental cost-effectiveness ratio to vary at the margin
analysts should avoid selective comparison of their results with the results from other studies
analysts should ensure that they consider the potential implications of uncertainty for the results of their analysis
interval estimates should accompany each point estimate presented
where sensitivity analysis is employed to estimate an interval, analysts should be comprehensive in their inclusion of all variables in the analysis
when reporting sensitivity analysis, analysts should be aware of the probabilistic nature of the reported range
when reporting patient level cost information, analysts should make more use of descriptive statistics
even when data are skewed, economic analyses should be based on means of distributions
when reporting statistical tests of cost differences, analysts should be aware that significance tests may be more powerful on a transformed scale but that confidence limits should be reported on the original scale
where patient level data on both cost and effect are available, the parametric approach based on Fieller's theorem or the non-parametric approach of bootstrapping should be employed to estimate a confidence interval for the cost-effectiveness ratio
sensitivity analysis has a continuing role in handling uncertainty not related to sampling variation
consideration should be given to using cost-effectiveness acceptability curves to present uncertainty in stochastic cost-effectiveness studies.

Recommendations for future research

Three main areas for future research arise from this review:

research into the appropriate reference case for the UK
research into the application of probabilistic sensitivity analysis methods
research into the willingness to pay for health gain and the likely value of a ceiling cost-effectiveness ratio appropriate for decision making, estimated from consumer surveys and implied through the application of cost-effectiveness databases.

Publication

Briggs AH, Gray AM. Handling uncertainty when performing economic evaluation of healthcare interventions. Health Technol Assessment 1999; 3(2).

NHS R&D HTA Programme

The overall aim of the NHS R&D Health Technology Assessment (HTA) programme is to ensure that high quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and work in the NHS. Research is undertaken in those areas where the evidence will lead to the greatest benefits to patients, either through improved patient outcomes or the most efficient use of NHS resources.

The Standing Group on Health Technology advises on national priorities for health technology assessment. Six advisory panels assist the Standing Group in identifying and prioritising projects. These priorities are then considered by the HTA Commissioning Board supported by the National Coordinating Centre for HTA.

This report is one of a series covering acute care, diagnostics and imaging, methodology, pharmaceuticals, population screening, and primary and community care. It was identified as a priority by the Methodology Panel and funded as project number 93/48/07.

The views expressed in this publication are those of the authors and not necessarily those of the Standing Group, the Commissioning Board, the Panel members or the Department of Health. The editors wish to emphasize that funding and publication of this research by the NHS should not be taken as implicit support for the recommendations for policy contained herein. In particular, policy options in the area of screening will be considered by the National Screening Committee. This Committee, chaired by the Chief Medical Officer, will take into account the views expressed here, further available evidence and other relevant considerations.

Reviews in Health Technology Assessment are termed 'systematic' when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

Series Editors:
Andrew Stevens, Ruairidh Milne, Ken Stein

Assistant Editor:
Melanie Corris

The editors have tried to ensure the accuracy of this report but cannot accept responsibility for any errors or omissions. They would like to thank the referees for their constructive comments on the draft document.

HTA Home Page | Details page for this publication | Publications listing | Publications search