D Harrison,¹* H Muskett,¹ S Harvey,¹ R Grieve,² J Shahin,¹ K Patel,¹ Z Sadique,² E Allen,² R Dybowski,³ M Jit,⁴ J Edgeworth,⁵ C Kibbler,⁶ R Barnes,⁷ N Soni⁸ and K Rowan¹

¹Intensive Care National Audit and Research Centre, London, UK
²London School of Hygiene and Tropical Medicine, London, UK
³Inferspace, Pinner, UK
⁴Health Protection Agency, London, UK
⁵St Thomas’ Hospital, London, UK
⁶Royal Free Hospital, London, UK
⁷Cardiff University, Cardiff, UK
⁸Chelsea and Westminster Hospital, London, UK

*Corresponding author

Background

There is increasing evidence that invasive fungal disease (IFD) is more likely to occur in non-neutropenic patients in critical care units. A number of randomised controlled trials (RCTs) have evaluated antifungal prophylaxis in non-neutropenic, critically ill patients. Despite heterogeneity in the patient groups studied, the RCTs have demonstrated a remarkably homogeneous effect of antifungal prophylaxis on the risk of proven IFD [relative risk (RR) 0.46, 95% confidence interval (CI) 0.31 to 0.68] and suggested a reduction in mortality (RR 0.76, 95% CI 0.59 to 0.97). Given that the effectiveness of antifungal prophylaxis has been demonstrated only in groups at high risk of IFD and that more widespread use of antifungal drugs may promote resistance and drive up costs, it is necessary to establish a method to identify and target antifungal prophylaxis at those patients at highest risk of IFD, who stand to benefit most from any antifungal prophylaxis strategy.

Several models for identifying patients at high risk of IFD have been proposed, although these are limited with regard to the populations included, thereby limiting their generalisability to a mixed UK critical care population. No models have previously been developed or validated in UK NHS adult critical care patients.

Objectives

The Fungal Infection Risk Evaluation (FIRE) Study had six objectives:

• to undertake a systematic literature review to identify risk factors for IFD
• to undertake data collection on risk factors and IFD in patients admitted to UK NHS adult general critical care units
• to develop, and internally validate, risk models for invasive Candida infection using both classical statistical methods and machine learning techniques
• to externally validate the risk models for invasive Candida infection
• to assess the cost-effectiveness of targeting antifungal prophylaxis to admissions identified as high risk, based on the risk models for invasive Candida infection
• to make recommendations for future research, based on value of information analysis.

Methods

Identification of risk factors for invasive fungal disease

A systematic review of the literature was undertaken. Electronic searches were performed to identify published, English-language articles that met the following criteria: (1) they evaluated either multiple risk factors, a scoring system or a clinical decision rule for IFD in critically ill patients; (2) the control group consisted of patients without IFD or any other systemic infection; and (3) they studied adult (> 18 years) humans. Data extracted included methods of development and validation and performance measures of risk models or clinical decision rules. Methodological quality of reporting for the eligible articles was assessed.

Data collection for risk factors and outcomes of invasive fungal disease

Risk factors identified from the systematic review were reviewed and refined in consultation with clinical experts to produce the final data set. Data were collected at three decision time points: on admission to the critical care unit; at the end of the first 24 hours; and at the end of the third calendar day. The primary outcome was IFD, defined as a blood culture or sample from a normally sterile site showing yeast/mould cells in a microbiological or histopathological report. For statistical and economic modelling, the primary outcome was invasive Candida infection, defined as IFD (as above) positive for Candida species. Outcomes data were collected until discharge from critical care or death.

All adult general critical care units in England, Wales and Northern Ireland participating in the Case Mix Programme were invited to take part and all adult general critical care units in Scotland. Staff in participating critical care units collected data according to precise rules and definitions on every consecutive patient admitted to their unit. Data were entered on to a dedicated, secure web-based data entry system. At the end of data collection, a reliability study was conducted to confirm that all cases of IFD were correctly diagnosed and recorded.

Development and validation of risk models for invasive Candida infection

The data set was divided into the following development and validation samples: (1) development sample – all admissions to a random sample of participating critical care units in England, Wales and Northern Ireland, July 2009 to December 2010; (2) random validation sample – all admissions to the remaining units in England, Wales and Northern Ireland; (3) temporal validation sample – all admissions to units in the development sample, January to March 2011; and (4) geographical validation sample – all admissions to units in Scotland. Logistic regression models were derived to model the risk of subsequently developing invasive Candida infection based on information available at the three time points. Candidate variables were identified and alternative approaches to modelling each individual risk factor were compared and evaluated in univariable analyses. All candidate variables were then included in a full multivariable model and the model was progressively simplified using backwards stepwise selection. Model discrimination was assessed with the c-index, equivalent to the area under the receiver operating characteristic curve, calibration by graphical plots of observed against expected risk, and overall fit by Brier’s score.

Bootstrapping was used to internally validate the final model at each time point and to estimate optimism-adjusted measures of discrimination and overall fit. The final model at each time point was evaluated in the three external validation samples.

The following alternative approaches to developing risk models using machine learning techniques were explored: feedforward neural networks (FFNNs); support vector machines; and random forests. Missing values were imputed using a combination of cold- and hot-deck imputation. Balanced pseudo-samples were created using the SMOTE data-rebalancing algorithm.

Economic modelling to assess the cost-effectiveness of prophylaxis based on the risk models for invasive Candida infection

The economic evaluation assessed the cost-effectiveness of alternative strategies to risk assessment followed by prophylaxis using the risk models developed for invasive Candida infection. Alternative treatment protocols for providing antifungal prophylaxis to patients identified as high risk (‘interventions’) were compared with providing no prophylaxis (‘current practice’). The treatment regimen evaluated followed current recommendations for 400 mg of fluconazole per day for 7 days. A decision-analytical approach to project lifetime cost-effectiveness was used. The decision model was populated with estimates of positive predictive value (the proportion of those identified as high risk who subsequently developed invasive Candida infection) and negative predictive value (the proportion of those identified as low risk who did not subsequently develop invasive Candida infection) from the risk models at each time point, and estimates of the effectiveness of antifungal prophylaxis from systematic reviews of published RCTs. A probabilistic sensitivity analysis was undertaken to recognise the sampling uncertainty surrounding the input parameters. The main structural assumptions were subjected to sensitivity analyses. Finally, the value of further research was established both overall and for specific parameters.

Results

Identification of risk factors for invasive fungal disease

Thirteen articles exploring risk factors, risk models or clinical decision rules for IFD in critically ill adult patients were identified. Of these, eight examined risk factors specifically, four developed risk models or clinical decision rules, and one evaluated a clinical decision rule.

The following risk factors were found in multiple studies to be significantly associated with IFD: surgery, total parenteral nutrition, fungal colonisation, renal replacement therapy, infection, mechanical ventilation, diabetes and acute severity scores. The risk model and clinical decision rule studies used all of these risk factors apart from mechanical ventilation and acute severity scores and, in addition, included pancreatitis and immunosuppressant use.

Risk factor definitions varied across studies, with many studies offering no definition at all. Risk factor selection process and modelling strategy also varied and no studies had an adequate sample size for multivariable analyses.

Data collection for risk factors and outcomes of invasive fungal disease

Data on 60,778 admissions to 96 adult general critical care units were collected between July 2009 and March 2011. The reliability study identified substantial over-reporting of IFD in the original data submissions, suggesting difficulty in correctly applying the IFD definitions. A large number of cases originally recorded as IFD were amended after verification from the local principal investigator that the original data were incorrect.

In total, 383 admissions (0.6%) were admitted with or developed IFD. The majority (94%) were infected with Candida species. The most common IFD infection site was blood [55%, followed by peritoneal fluid (25%) and pleural fluid (10%)]. The incidence of IFD identified in unit was 4.7 cases per 1000 admissions overall, 3.2 per 1000 for unit-acquired IFD, and 3.5 per 1000 for IFD in blood.

Development and validation of risk models for invasive Candida infection

The data set was divided into development and validation samples as follows: development sample – 39,685 admissions to 70 units; random validation sample – 4669 admissions to 10 units; temporal validation sample – 11,051 admissions to 66 units; and geographic validation sample – 5373 admissions to 16 units. The final risk model at admission included the following variables: admission for presurgical preparation; surgery within up to 7 days prior to admission (elective/scheduled with no unexpected complications; elective/scheduled with unexpected complications; emergency/urgent; no surgery); pancreatitis; number of catheters in central veins; number of drains; enteral feeding tube; and number of samples positive for fungal colonisation. The final risk model at 24 hours included surgery within up to 7 days prior to admission (elective/scheduled, emergency/urgent, no surgery), pancreatitis, number of catheters in central veins, number of drains, lowest systolic blood pressure, highest heart rate, and number of samples positive for fungal colonisation. The final risk model at the end of calendar day 3 included pancreatitis, number of catheters in central veins, number of drains, highest heart rate, and number of samples positive for fungal colonisation. The risk model at admission had fair discrimination (c-index 0.705). Discrimination improved at 24 hours (c-index 0.823) and this was maintained at the end of calendar day 3 (c-index 0.835). Despite the huge sample size, the low rate of invasive Candida infection made robust statistical modelling difficult. Consequently, the resulting events per variable of the models was low (five for the full model at admission). This leaves the possibility that the models may have been overfitted, and this may contribute to the drop in model performance when assessed in the validation samples (c-index 0.655, 0.732 and 0.709 for the three models in the full validation sample). Model performance was worst when applied in the geographical validation sample, suggesting that particular care should be taken in transferring the models to different geographical settings.

Problems with local minima prevented the application of FFNNs, and a number of technical issues with the application of support vector machines were unable to be resolved. The random forest approach was therefore preferred, and this revealed a number of possible risk factors for invasive Candida infection and was seen to be a fairly accurate predictor within the balanced pseudo-samples created for model development (out-of-bag estimated overall misclassification rates for a random forest of 100 trees were 4.13%, 2.86% and 4.98%, respectively, for the three models).

Economic modelling to assess the cost-effectiveness of prophylaxis, based on the risk models for invasive Candida infection

Risk assessment and prophylaxis led to higher costs than current practice. However, prophylaxis was predicted to slightly reduce mean hospitalisation costs. The strategies with risk thresholds of 0.5% and 1% had higher mean total costs than with a risk threshold of 2% as they involved providing prophylaxis to a larger proportion of patients.

The incremental analysis showed that irrespective of the risk threshold, the incremental quality-adjusted life-years of the prophylaxis strategies compared with current practice were positive but small compared with the incremental costs. Incremental net benefits of each prophylaxis strategy compared with current practice were all negative. Cost-effectiveness acceptability curves showed that at each risk threshold, current practice was the strategy most likely to be cost-effective.

Across all parameters in the decision model, the results indicated that the value of further research for the whole population of interest might be high relative to the research costs, and that the greatest value is for the higher risk threshold, i.e. it is at this threshold that the recommendation, based on current evidence, to adopt current practice is most uncertain.

Conclusions

Implications for health care

The results of the FIRE Study, derived from a highly representative sample of adult general critical care units across the UK, indicated a low incidence of IFD among non-neutropenic, critically ill adult patients. However, IFD, although rare, was associated with substantially higher mortality, more intensive organ support and longer length of stay within both the critical care unit and acute hospital settings.

Risk modelling using classical statistical methods produced relatively simple risk models, and associated clinical decision rules, that provided acceptable discrimination for identifying patients at ‘high risk’ of invasive Candida infection but care should be taken when translating the models to a different health care system/setting.

Results of the economic model suggested that the current most cost-effective treatment strategy for prophylactic use of systemic antifungal agents among non-neutropenic, critically ill adult patients admitted to NHS adult general critical care units is a strategy of no risk assessment and no antifungal prophylaxis.

Structural sensitivity analysis indicated that reducing the dose of fluconazole (to one-quarter of that currently recommended), if equally effective, would result in the most cost-effective strategy being to assess risk at the end of calendar day 3 following admission and providing antifungal prophylaxis if the predicted risk exceeded 2%. However, it is unlikely that such a reduction in dose would be considered acceptable.

Recommendations for research

Recommendation 1: Further research should be conducted to inform the long-term survival, including quality and costs of survival, for the population of patients admitted to UK adult general critical care units.

Recommendation 2: Future research into treatment strategies for selecting patients for antifungal prophylaxis should consider combining clinical risk estimates, such as those from the FIRE Study risk models, with novel diagnostic tests based on biomarkers.

Recommendation 3: Further research should be considered to inform estimates of the positive and negative predictive values of the FIRE Study risk models among non-neutropenic, critically ill adult patients admitted to UK adult general critical care units.

Recommendation 4: Further research should be considered to inform estimates of baseline risk of IFD and associated outcomes among non-neutropenic, critically ill adult patients admitted to UK adult general critical care units.

Recommendation 5: Results of recommendations 1, 3 and 4 (above) should be re-evaluated for their impact on the decision model and value of information analyses.

Recommendation 6: Further research into machine learning techniques should be considered to establish whether or not current barriers to their implementation at the bedside can be overcome.

Funding

Funding for this study was provided by the Health Technology Assessment programme of the National Institute for Health Research.

Publication

Harrison D, Muskett H, Harvey S, Grieve R, Shahin J, Patel K, et al. Development and validation of a risk model for identification of non-neutropenic, critically ill adult patients at high risk of invasive Candida infection: the Fungal Infection Risk Evaluation (FIRE) Study. Health Technol Assess 2013;17(3).

NIHR Health Technology Assessment programme

The Health Technology Assessment (HTA) programme, part of the National Institute for Health Research (NIHR), was set up in 1993. It produces high-quality research information on the effectiveness, costs and broader impact of health technologies for those who use, manage and provide care in the NHS. ‘Health technologies’ are broadly defined as all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care.

The research findings from the HTA programme directly influence decision-making bodies such as the National Institute for Health and Clinical Excellence (NICE) and the National Screening Committee (NSC). HTA findings also help to improve the quality of clinical practice in the NHS indirectly in that they form a key component of the ‘National Knowledge Service’.

The HTA programme is needs led in that it fills gaps in the evidence needed by the NHS. There are three routes to the start of projects.

First is the commissioned route. Suggestions for research are actively sought from people working in the NHS, from the public and consumer groups and from professional bodies such as royal colleges and NHS trusts. These suggestions are carefully prioritised by panels of independent experts (including NHS service users). The HTA programme then commissions the research by competitive tender.

Second, the HTA programme provides grants for clinical trials for researchers who identify research questions. These are assessed for importance to patients and the NHS, and scientific rigour.

Third, through its Technology Assessment Report (TAR) call-off contract, the HTA programme commissions bespoke reports, principally for NICE, but also for other policy-makers. TARs bring together evidence on the value of specific technologies.

Some HTA research projects, including TARs, may take only months, others need several years. They can cost from as little as £40,000 to over £1 million, and may involve synthesising existing evidence, undertaking a trial, or other research collecting new data to answer a research problem.

The final reports from HTA projects are peer reviewed by a number of independent expert referees before publication in the widely read journal series Health Technology Assessment.

Criteria for inclusion in the HTA journal series

Reports are published in the HTA journal series if (1) they have resulted from work for the HTA programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

The research reported in this issue of the journal was funded by the HTA programme as project number 07/29/01. The contractual start date was in May 2008. The draft report began editorial review in November 2011 and was accepted for publication in May 2012. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the HTA programme or the Department of Health.

Editor-in-Chief: Professor Tom Walley CBE
Series Editors: Dr Martin Ashton-Key, Professor Aileen Clarke, Dr Peter Davidson, Ms Tara Lamont, Dr Tom Marshall, Professor William McGuire, Professor Jane Norman, Professor John Powell, Professor James Raftery, Dr Rob Riemsma, Professor Helen Roberts, Professor Helen Snooks and Professor Ken Stein