HTA Home Page | Details page for this publication | Publications listing | Publications search

Health Technology Assessment 2001; Vol. 5: No. 12

Executive summary

View/Download full monograph in Adobe Acrobat format (689 kbytes)
View/Download this 4-page summary in Adobe Acrobat format  (suitable for printing)

Statistical assessment of the learning curves of health technologies

CR Ramsay¹
AM Grant^1*
SA Wallace¹
PH Garthwaite²
AF Monk³
IT Russell³

¹ Health Services Research Unit, University of Aberdeen, Aberdeen, UK
² Open University, Milton Keynes, UK
³ University of York, York, UK

^* Corresponding author

Objectives

• To describe systematically studies that directly assessed the learning curve effect of health technologies.

• Systematically to identify ‘novel’ statistical techniques applied to learning curve data in other fields, such as psychology and manufacturing.

• To test these statistical techniques in data sets from studies of varying designs to assess health technologies in which learning curve effects are known to exist.

Methods

Study selection

Health technology assessment literature review

For a study to be included, it had to include a formal analysis of the learning curve of a health technology using a graphical, tabular or statistical technique.

Non-health technology assessment literature search

For a study to be included, it had to include a formal assessment of a learning curve using a statistical technique that had not been identified in the previous search.

Data sources

Six clinical and 16 non-clinical biomedical databases were searched. A limited amount of handsearching and scanning of reference lists was also undertaken.

Data extraction

Health technology assessment literature review

A number of study characteristics were abstracted from the papers such as study design, study size, number of operators and the statistical method used.

Non-health technology assessment literature search

The new statistical techniques identified were categorised into four subgroups of increasing complexity: exploratory data analysis; simple series data analysis; complex data structure analysis, generic techniques.

Testing of statistical methods

Some of the statistical methods identified in the systematic searches for single (simple) operator series data and for multiple (complex) operator series data were illustrated and explored using three data sets. The first was a case series of 190 consecutive laparoscopic fundoplication procedures performed by a single surgeon; the second was a case series of consecutive laparoscopic cholecystectomy procedures performed by ten surgeons; the third was randomised trial data derived from the laparoscopic pŒocedure arm of a multicentre trial of groin hernia repair, supplemented by data from non-randomised operations performed during the trial.

Results

Health technology assessment literature review

Of 4571 abstracts identified, 272 (6%) were later included in the study after review of the full paper. Some 51% of studies assessed a surgical minimal access technique and 95% were case series. The statistical method used most often (60%) was splitting the data into consecutive parts (such as halves or thirds), with only 14% attempting a more formal statistical analysis. The reporting of the studies was poor, with 31% giving no details of data collection methods.

Non-health technology assessment literature search

Of 9431 abstracts assessed, 115 (1%) were deemed appropriate for further investigation and, of these, 18 were included in the study. All of the methods for complex data sets were identified in the non-clinical literature. These were discriminant analysis, two-stage estimation of learning rates, generalised estimating equations, multilevel models, latent curve models, time series models and stochastic parameter models. In addition, eight new shapes of learning curves were identified.

Testing of statistical methods

No one particular shape of learning curve performed significantly better than another. The performance of ‘operation time’ as a proxy for learning differed between the three procedures. Multilevel modelling using the laparoscopic cholecystectomy data demonstrated and measured surgeon-specific and confounding effects. The inclusion of non-randomised cases, despite the possible limitations of the method, enhanced the interpretation of learning effects.

Conclusions

Health technology assessment literature review

The statistical methods used for assessing learning effects in health technology assessment have been crude and the reporting of studies poor.

Non-health technology assessment literature search

A number of statistical methods for assessing learning effects were identified that had not hitherto been used in health technology assessment. There was a hierarchy of methods for the identification and measurement of learning, and the more sophisticated methods for both have had little if any use in health technology assessment. This demonstrated the value of considering fields outside clinical research when addressing methodological issues in health technology assessment.

Testing of statistical methods

It has been demonstrated that the portfolio of techniques identified can enhance investigations of learning curve effects.

Implications and recommendations

For health technology assessment

• A change over time in the performance of a technology because of learning complicates evaluation and impedes rigorous evaluation.

• Useful parameters for describing learning in health technology assessment are the rate and length of learning and the final skill level.

• Reliable assessment of learning effects is most likely to come from prospectively collected data on multiple operators or institutions.

• The experience of the operator should be described each time the procedure is performed. This is particularly important in circumstances, such as randomised trials, in which the technology may have parallel use outside the trial.

• Collection of non-randomised data alongside a randomised controlled trial may, despite possible limitations, aid the interpretation of learning effects.

• Reports of studies of learning should, as a minimum, describe the number and experience of the operators, the data source, the proportion of procedures performed by individual operators and the level of care.

• Proxy measures of learning have advantages and limitations, and finding a suitable measure can be difficult.

• Investigators should consider and adjust for any confounding factors.

• The simplest methods within the hierarchies described in this report should be used in a parsimonious way.

• When there are multiple operators, a method should be used which takes into account the hierarchical nature of the data.

For further research

• Further empirical testing of the techniques identified is required. The generalisability of the various shapes and methods that were identified needs to be assessed for a variety of health technologies.

• Methods for estimation of the time taken to reach an asymptote should be explored further.

• Variables that are good proxies for learning need to be identified.

• Relatively rare, dichotomous outcomes are often the best measures of performance but are currently the least tractable to analysis. Further methodological research is needed to address this issue.

• Further empirical work is required to identify the optimal method for assessing learning curves within randomised controlled trials.

• The impact of learning curve factors on economic evaluations should be explored.

• Appropriate prospective data collection should be built into future evaluations.

• A theory-based approach to learning should be investigated in the context of health technology assessment.

• Parallels between learning curve and quality assurance issues should be explored.

Publication

Ramsay CR, Grant AM, Wallace SA, Garthwaite PH, Monk AF, Russell IT. Statistical assessment of the learning curves of health technologies. Health Technol Assess 2001; 5(12).

NHS R&D HTA Programme

The NHS R&D Health Technology Assessment (HTA) Programme was set up in 1993 to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and provide care in the NHS.

Initially, six HTA panels (pharmaceuticals, acute sector, primary and community care, diagnostics and imaging, population screening, methodology) helped to set the research priorities for the HTA Programme. However, during the past few years there have been a number of changes in and around NHS R&D, such as the establishment of the National Institute for Clinical Excellence (NICE) and the creation of three new research programmes: Service Delivery and Organisation (SDO); New and Emerging Applications of Technology (NEAT); and the Methodology Programme.

Although the National Coordinating Centre for Health Technology Assessment (NCCHTA) commissions research on behalf of the Methodology Programme, it is the Methodology Group that now considers and advises the Methodology Programme Director on the best research projects to pursue.

The research reported in this monograph was funded as project number 96/25/02.

The views expressed in this publication are those of the authors and not necessarily those of the Methodology Programme, HTA Programme or the Department of Health. The editors wish to emphasize that funding and publication of this research by the NHS should not be taken as implicit support for any recommendations made by the authors.

Criteria for inclusion in the HTA monograph series
Reports are published in the HTA monograph series if (1) they have resulted from work commissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

Methodology Programme Director: Professor Richard Lilford
HTA Programme Director: Professor Kent Woods
Series Editors: Professor Andrew Stevens, Dr Ken Stein, Professor John Gabbay
and Dr Ruairidh Milne
Monograph Editorial Manager: Melanie Corris

The editors and publisher have tried to ensure the accuracy of this report but do not accept liability for damages or losses arising from material published in this report. They would like to thank the referees for their constructive comments on the draft document.

©2001 Crown Copyright

HTA Home Page | Details page for this publication | Publications listing | Publications search