Health Technology Assessment 2004; Vol 8: number 49
View/Download full monograph in Adobe Acrobat format
4-page summary in Adobe Acrobat format (suitable for printing)
Generalisability in economic evaluation studies in healthcare: a review and case studies
MJ Sculpher,1* FS Pang,1 A Manca,1 MF Drummond,1 S Golder,2 H Urdahl,1
LM Davies3 and A Eastwood2
1 Centre for Health Economics, University of York, UK
2 Centre for Reviews and Dissemination, University of York, UK
3 Manchester Medical School, Manchester University, UK
* Corresponding author
Given the increasing need for economic
evidence to inform the resource allocation decisions of a range of
decision-makers and in many jurisdictions, there is interest in the
generalisability of economic evaluations, that is, the extent to which the
results of a study based on measurement in a particular patient population
and/or a specific context hold true for another population and/or in a
different context. The context
which is the primary focus of this report is the location in which the study
was undertaken and/or the decision-maker
for whom the study was undertaken. The focus of this report is economic evaluation
as applied to health services.
Aims and objectives
The aim of the project was to review, and
to develop further, the methods used to assess and to increase the
generalisability of economic evaluation studies.
The specific objectives were to conduct:
- A systematic review of methods
literature on generalisability relating to economic evaluation to identify
factors causing variability in cost-effectiveness between locations and over
time, and the extent of that variability.
- A systematic review of methods literature
on economic evaluation relating to available methods to assess variability
between locations and over time.
- A systematic review of applied economic
evaluation studies undertaken alongside multilocation trials to describe how
studies have assessed and reported generalisability and variability in results
- A series of case studies involving the
secondary analysis of cost-effectiveness analyses undertaken alongside
multilocation trials to explore the use of multilevel modelling to assess
variability in cost-effectiveness between locations.
- A structured review of economic
evaluations based on decision analytic models in the field of osteoporosis to
describe how studies have made their analyses relevant to particular decision-makers/jurisdictions
and assessed how results might vary across locations.
- A case study of a decision analytic
model to illustrate methods to estimate cost-effectiveness for the NHS based on
data partly collected in non-UK locations.
For Objectives 1 and 2 above, methodological studies relating to economic evaluation in healthcare were
searched. This included electronic searches of a range of databases, including
PREMEDLINE, MEDLINE, EMBASE and EconLit, and manual searches of key journals.
Similar methods were used for Objectives 3 and 5 to identify applied
economic studies. The case studies (Objectives 4 and 6) involved highlighting
specific features of previously published economic studies related to
generalisability and location-related variability. In the case of Objective 4,
the case-study was based on the secondary analysis of three economic studies
using data from randomised trials.
Variability in cost-effectiveness by time and place
- The factor most frequently cited as generating variability
in economic results between locations was the unit costs associated with
- Some of the most frequently cited factors are as much
associated with the measurement of effectiveness as with cost-effectiveness (e.g. the
artificial characteristics of trials and patient case mix).
- No studies were identified which explicitly considered
factors causing variability in the results of economic studies over time.
- Several authors have shown important variations between
locations in the volume and cost of resource use and in cost-effectiveness.
Methods to assess variability in cost-effectiveness by time and place
- In the context of
studies based on the analysis of patient-level data, regression analysis has
been advocated as a means of looking at variability in economic results across
locations. These methods have generally accepted that some components of
resource use and outcomes are exchangeable across locations whereas others are
- Recent studies have
also explored, in cost-effectiveness analysis, the use of tests of
heterogeneity similar to those used in clinical evaluation in trials.
- The decision
analytic model has been the main means by which cost-effectiveness has been
adapted from trial to non-trial locations. Most models have focused on changes
to the cost side of the analysis, but it is clear that the effectiveness side
may also need to be adapted between locations.
- The review failed to
identify a major literature on variability in cost-effectiveness over time, although
an emerging literature using Bayesian decision theory may be of value.
Dealing with variability by location in economic studies alongside multilocation trials
- There have been
weaknesses in some aspects of the reporting in applied cost-effectiveness
studies. These may limit decision-makers ability to judge the relevance of a
study to their specific situations.
- There was little use
of the statistical approaches identified in the methods review to assess
variability by location.
- The case study
demonstrated the potential value of multilevel modelling (MLM). Where
clustering exists by location (e.g. centre or country), MLM can facilitate
correct estimates of the uncertainty in cost-effectiveness results.
- MLM also provides a means of estimating location-specific cost-effectiveness.
- The use of location-specific covariates in MLM can explain some of the variation in cost-effectiveness.
- An important policy
issue is raised by this work: the extent to which location-specific estimates
of incremental net benefit are useful to decision makers.
Use of decision analytic models to provide location-specific estimates of cost-effectiveness
- The review of
applied economic studies based on decision analytic models showed that few
studies were explicit about their target decision-maker(s)/jurisdictions.
- The studies in the
review generally made more effort to ensure that their cost inputs were
specific to their target jurisdiction than their effectiveness parameters.
sensitivity analysis was the main way of dealing with uncertainty in the
models, although few studies looked explicitly at variability between
- The modelling case
study illustrated how effectiveness and cost data can be made
location-specific. In particular, on the effectiveness side, the example showed
the separation of location-specific baseline events and pooled estimates of
relative treatment effect, where the latter are assumed exchangeable across
Economic evaluation using patient-level data
- At the design stage
of a study, selection of study sites should ideally focus on those that are
representative of the jurisdiction(s) for which economic data are required.
- There is value in
collecting data on the characteristics of trial centres which could be used as
covariates in regression models.
- The patients
included in studies should reflect the normal clinical caseload, but it is
important to collect a number of patient-level variables that could be used as
- Resource use data
(e.g. hospital days) should be reported separately from the unit costs of those
- MLM should be
considered as a means of assessing the degree of clustering in cost and
effectiveness data within trial locations. If clustering is extensive, MLM can
reflect this characteristic at the analysis stage and generate
location-specific estimates of cost-effectiveness.
- There remains an
important role for sensitivity analysis in exploring the implications of
variation in some parameters (e.g. unit costs and preference values).
- Reporting more
information on the centres/countries in a study can assist decision-makers in
interpreting the relevance of results to their situation.
Economic evaluation using decision analytic modelling
- Given the focus on a
decision, any analysis should be clear about the specification of the
decision problem and the relevant decision-maker(s) and jurisdiction(s).
- The overall
analytical approach, model structure and data inputs should be appropriate to
the relevant decision-maker(s).
- Where several
sources of data exist for a particular parameter, these should be pooled in
such a way that the uncertainty relating to their precision and possible
heterogeneity (including that related to location) is reflected in the model.
- It is important to
distinguish parameter uncertainty from variability or heterogeneity, where the
latter is concerned with how parameter estimates vary across contexts.
analysis, where data inputs are incorporated as random variables, is the
appropriate means of handling parameter uncertainty.
- When a model is
targeted at more than one decision-maker/jurisdiction, an important aspect of
the analysis is to assess the variability in results between locations, for example, using sensitivity or scenario analysis.
A large number of factors are mentioned in
the literature that might be expected to generate variation in the
cost-effectiveness of healthcare interventions across locations. Several papers
have demonstrated differences in the volume and cost of resource use between
locations, but few studies have looked at variability in outcomes.
In applied trial-based cost-effectiveness
studies, few studies provide sufficient evidence for decision-makers to
establish the relevance or to adjust the results of the study to their location
of interest. Very few studies utilised statistical methods formally to assess
the variability in results between locations. In applied economic studies based
on decision models, most studies either stated their target
decision-maker/jurisdiction or provided sufficient information from which this
could be inferred. There was a greater tendency to ensure that cost inputs were
specific to the target jurisdiction than clinical parameters.
Methods to assess generalisability and
variability in economic evaluation studies have been discussed extensively in
the literature relating to both trial-based and modelling studies.
Regression-based methods are likely to offer a systematic approach to
quantifying variability in patient-level data. In particular, MLM has the
potential to facilitate estimates of cost-effectiveness which both reflect the
variation in costs and outcomes between locations and also enable the
consistency of cost-effectiveness estimates between locations to be assessed
directly. Decision analytic models will retain an important role in adapting
the results of cost-effectiveness studies between locations.
Summary of recommendations for further research
Drawing on the material in this report, it
is possible to summarise some important areas for further research. As far as
possible, these have been placed in priority order.
- The development of
methods of evidence synthesis which model the exchangeability of data across
locations and allow for the additional uncertainty in this process. These
methods should relate to all parameters relevant to economic evaluation.
- Assessment of
alternative approaches to specifying multilevel models to the analysis of
cost-effectiveness data alongside multilocation randomised trials.
- Identification of a
range of appropriate covariates relating to locations (e.g. hospitals) in
- Further assessment
of the role of econometric methods (e.g. selection models) for
cost-effectiveness analysis alongside observational datasets, and to increase
the generalisability of randomised trials.
Sculpher MJ, Pang FS, Manca A, Drummond MF, Golder S, Urdahl H, et al. Generalisability in economic evaluation studies in healthcare: a review and
case studies. Health Technol Assess 2004;8(49).
NHS R&D HTA Programme
The research findings
from the NHS R&D Health Technology Assessment (HTA) Programme directly
influence key decision-making bodies such as the National Institute for
Clinical Excellence (NICE) and the National Screening Committee (NSC) who rely
on HTA outputs to help raise standards of care. HTA findings also help to
improve the quality of the service in the NHS indirectly in that they form a
key component of the National Knowledge Service that is being developed to
improve the evidence of clinical practice throughout the NHS.
The HTA Programme was
set up in 1993. Its role is to ensure that high-quality research information on
the costs, effectiveness and broader impact of health technologies is produced
in the most efficient way for those who use, manage and provide care in the
NHS. Health technologies are broadly defined to include all interventions
used to promote health, prevent and treat disease, and improve rehabilitation
and long-term care, rather than settings of care.
The HTA programme
commissions research only on topics where it has identified key gaps in the
evidence needed by the NHS. Suggestions for topics are actively sought from
people working in the NHS, the public, consumer groups and professional bodies
such as Royal Colleges and NHS Trusts.
are carefully considered by panels of independent experts (including consumers)
whose advice results in a ranked list of recommended research priorities. The
HTA Programme then commissions the research team best suited to undertake the
work, in the manner most appropriate to find the relevant answers. Some
projects may take only months, others need several years to answer the research
questions adequately. They may involve synthesising existing evidence or
designing a trial to produce new evidence where none currently exists.
its Technology Assessment Report (TAR) call-off contract, the HTA Programme is
able to commission bespoke reports, principally for NICE, but also for other
policy customers, such as a National Clinical Director. TARs bring together
evidence on key aspects of the use of specific technologies and usually have to
be completed within a limited time period.
Criteria for inclusion in the HTA monograph series
Reports are published in the HTA monograph
series if (1) they have resulted from work commissioned for the HTA Programme,
and (2) they are of a sufficiently high scientific quality as assessed by the
referees and editors.
Reviews in Health Technology Assessment are termed systematic when the
account of the search, appraisal and synthesis methods (to minimise biases and
random errors) would, in theory, permit the replication of the review by
The research reported
in this monograph was commissioned by the HTA Programmeas project number 98/22/05.
As funder, by devising a commissioning brief, the HTA Programme specified the
research question and study design. The authors have been wholly responsible
for all data collection, analysis and interpretation and for writing up their
work. The HTA editors and publisher have tried to ensure the accuracy of the
authors report and would like to thank the referees for their constructive
comments on the draft document. However, they do not accept liability for
damages or losses arising from material published in this report.
The views expressed in this publication are those of the authors and not necessarily those of the HTA
Programme or the Department of Health.
Editor-in-Chief: Professor Tom Walley
Series Editors: Dr Peter Davidson, Professor John Gabbay, Dr Chris Hyde, Dr Ruairidh Milne, Dr Rob Riemsma and Dr Ken Stein
Managing Editors: Sally Bailey and Caroline Ciupek
© 2004 Crown Copyright