Executive summary of HTA journal title
Health Technol Assess 2011;15(37):1–160
Adaptive e-learning to improve dietary behaviour: a systematic review and cost-effectiveness analysis.
View/Download full text in Adobe Acrobat format (8.1MB)
View/Download this summary in Adobe Acrobat format (suitable for printing)
J Harris,1 L Felix,1 A Miners,2 E Murray,3 S Michie,4 E Ferguson,1 C Free,1 K Lock,2 J Landon5 and P Edwards1*
1Faculty of Epidemiology and Population Health, London School of Hygiene and Tropical Medicine, London, UK
2Faculty of Public Health and Policy, London School of Hygiene and Tropical Medicine, London, UK
3Research Department of Primary Care and Population Health, University College London, London, UK
4Research Department of Clinical, Educational & Health Psychology, University College London, London, UK
5National Heart Forum, London, UK
The composition of habitual diets is associated with adverse or protective effects on health. Consequently, UK public health policy strongly advocates dietary change for the improvement of population health and emphasises the importance of individual empowerment to improve health.
A new and evolving area in the promotion of dietary behavioural change is ‘e-learning’, the use of interactive electronic media to facilitate teaching and learning on a range of issues including health. The high level of accessibility, combined with emerging advances in computer processing power, data transmission and data storage, makes interactive e-learning a potentially powerful and cost-effective medium for improving dietary behaviour. E-learning also has a number of potential advantages compared with traditional approaches for the promotion of dietary behaviour change, such as the possibility of tailoring to individual circumstances; translating complex information through video, graphics and audio systems; and potential cost savings on face-to-face interventions involving health-care practitioners. Some evidence that individualised, tailored e-learning approaches are more effective than traditional non-tailored interventions has given them a promising lead in health education.
The aims of this systematic review were to assess the effectiveness and cost-effectiveness of adaptive e-learning for improving dietary behaviours. The specific objectives were to:
- describe the range of e-learning technologies in use for promoting dietary behavioural change
- evaluate the effectiveness of interactive e-learning in terms of improvement in dietary behaviour and clinical outcomes
- analyse the e-learning interventions in order to determine the components contributing to effects of e-learning interventions for dietary behaviour change
- investigate potential explanations of dietary behaviour change and mechanisms of action
- evaluate cost-effectiveness compared with current standard interventions, and estimate the likely budget impact in England and Wales.
We included all randomised controlled trials (RCTs) of effectiveness of e-learning in adolescents or adults to promote dietary behavioural change and all clinical conditions in which dietary advice plays a major part in case management. Interventions were included if they were interactive computer software programs that tailored output according to user input (including interventions where users enter personal data, or make choices about information, that alter pathways within programs to produce tailored material and feedback that is personally relevant).
Primary outcomes were measures of dietary behaviours, including estimated intakes or changes in intake of energy, nutrients, dietary fibre, foods or food groups. Secondary outcome measures were clinical outcomes that would be expected to respond to changes in dietary behaviours (e.g. anthropometric status and blood biochemistry). We also sought data on the costs of providing the intervention, any unintended adverse consequences of the interventions, process outcomes (e.g. usage) and data relating to potential cognitive and emotional mediators of dietary behaviour.
Searching, screening and data extraction
Eligible studies were identified by searches of 11 electronic bibliographic databases, trial registers for ongoing and recently completed trials, inspection of the reference lists of all included studies and previously published reviews, and by contact with authors of included studies. There were no restrictions by language. Searches covered the period January 1990 to November 2009.
Two review authors independently examined the titles, abstracts and keywords of electronic records according to the eligibility criteria above. The results of this initial screening were cross-referenced between the two review authors and full-text records obtained for all potentially relevant reports of trials. These potentially eligible trials went through a secondary screening by each reviewer using a screening form based on the eligibility criteria for final inclusion in the systematic review, with disagreements resolved by discussion with a third author.
Two review authors extracted relevant data into a Microsoft Access 2007 database (Microsoft Corporation, Redmond, WA, USA) specifically designed for the review. Two measures of methodological quality were used in the review: the Cochrane Collaboration’s risk of bias assessment and the Effective Public Health Practice Project (EPHPP) quality assessment. The dietary assessment tools and techniques used to estimate dietary behaviour were critically examined in terms of quality.
Analysis of effectiveness
For studies that reported the same outcome, we pooled the results using a random effects model, with weighted mean differences (WMDs), and calculated 95% confidence intervals (CIs) and two-sided p-values for each outcome. When outcomes were assessed more than once during follow-up, the final assessment was used in analysis. We assessed evidence for selection bias using Egger’s test for small study effects. Heterogeneity among the trial results was assessed using both a chi-squared test and the I2 statistic (we considered I2 > 50% to reflect ‘substantial heterogeneity’). We conducted sensitivity analyses to investigate possible sources of heterogeneity including study quality and sociodemographic factors that could act as effect modifiers. Causes of heterogeneity and subgroup effects were assessed using random effects meta-analysis. This was implemented in Stata (StataCorp LP, College Station, TX, USA) using the ‘metareg’ command and including trial characteristics as covariates. All statistical analysis was conducted using Stata statistical software version 11.
The intention was to estimate cost-effectiveness in two ways: through a systematic literature review and by building a de novo decision model to assess the cost-effectiveness of a ‘generic’ e-learning device compared with dietary advice delivered by a health-care professional. However, no perfectly fitting published economic evaluations were identified; thus, the results from the literature review were instead primarily used to inform the model design. The key assumption within the model was that the interventions were designed to promote weight loss in already obese people with a body mass index (BMI) of ≥ 30 kg/m2. The evaluation was performed from a UK NHS cost perspective and outcomes were expressed as quality-adjusted life-years (QALYs). The estimate of relative treatment effect was derived from the systematic review of effectiveness, and the model was based on discrete event simulation techniques referred to as the ‘E-Learning Economic Evaluation Model’ (E-LEEM).
A total of 36,379 titles were initially identified by the electronic searches, of which 2977 were duplicates and were removed. The remaining 33,402 records were screened and 33,129 records were excluded because they did not meet the inclusion criteria. Full-text reports were sought for 273 potentially eligible studies, of which 233 studies were excluded after inspection of the full report. Three eligible studies were identified through searching reference lists of the included studies, yielding a total of 43 studies for inclusion in the review.
Description of included studies
Of the 43 included studies, one was a crossover trial and two were cluster-randomised trials, whereas the rest were parallel-group RCTs. The majority of interventions evaluated sought to reduce fat intake (28 interventions) and/or to increase fruit and vegetable intake (21 interventions). Other interventions sought to increase fibre intake, reduce overall energy intake, reduce or maintain weight, or reduce or maintain BMI.
All e-learning interventions were delivered in high-income countries: the majority were delivered in the USA (29 interventions), the Netherlands (five studies) and Belgium (three studies). Many interventions were offered over the internet or via a mobile device and without a specific setting. Of those that were delivered in a specific setting, nine were designed to be delivered in the home, eight in the workplace, three in community centres, four in schools/colleges and two in supermarkets. Many study reports did not provide information on the intended intensity or frequency of use; of those that did, intended intensity of interventions varied between a single session to weekly interaction with the intervention over 1 year.
A majority of interventions involved the presentation of nutrition, health and lifestyle information (31 studies), the entering of food consumption data (28 studies), and the presentation of personalised feedback on food and nutrient consumption (21 studies). Other interventions focused on the setting of goals and providing feedback on attainment of these goals (15 studies). The most commonly used behavioural change techniques reported were goal setting (behaviour) (14 studies); provide feedback on performance (14 studies); provide information on consequences of behaviour in general (14 studies) or to the individual (11 studies); barrier identification/problem solving (13 studies); prompt self-monitoring of behaviour (12 studies); provide instruction on how to perform the behaviour (12 studies); prompt review of behavioural goals (11 studies); and plan social support/social change (10 studies).
Of the 23 studies providing details of inclusion criteria, 11 offered the intervention only to those with BMI > 25 kg/m2; 10 required participants to be generally healthy and free of diagnosed disease; one trialled the intervention in patients with diabetes; and one required participants to have at least one diagnosed risk factor for cardiovascular disease. Three studies included adolescents aged < 18 years. Of studies targeting adults, the majority included participants with an average age of 40–49 years. One study included adults with an average age > 60 years.
Twenty-two studies measured total fat intake and eight studies measured saturated fat intake (either as grams consumed or as percentage of total energy consumed). Twenty-two studies assessed servings of fruit and/or vegetables consumed per day. Other food and nutrient outcomes measured included energy intake (nine studies) and fibre intake (five studies). In terms of clinical outcomes, 11 studies measured BMI, 14 studies measured weight or weight loss, four studies measured cholesterol, four studies assessed triglycerides and three studies assessed blood pressure. Four studies concentrated on the avoidance of eating disorders and used scales of eating disorder risk alongside measures of BMI and/or weight.
The Cochrane assessment was at ‘low risk of bias’ in 27 studies for methods of sequence generation; 18 studies for methods of allocation concealment; 15 studies for methods of blinding; 22 studies for addressing incomplete outcome data; 11 studies for not demonstrating selective outcome reporting; and 26 studies for providing a conflict of interest statement. The EPHPP assessment was ‘strong’ in one study for selection bias criteria; 43 studies for study design; 36 studies for adjustment for confounders; five studies for criteria on blinding; 21 studies for data collection methods; and 24 studies for attrition criteria. Thirteen studies were rated overall as ‘moderate’ and 30 studies were rated overall as ‘weak’; none was rated ‘strong’ overall.
Studies using 24-hour recall (two studies), or a combination of two or more dietary assessment methods (11 studies), especially two independent dietary assessment methods (two studies), had the strongest dietary assessment methods for measuring dietary change. Those studies that used a prospective dietary assessment method (e.g. multiple day diet record) alone are valid for measuring dietary change, as long as a compliance bias did not result in an overestimation of the intervention effect and dietary practices were not simplified to reduce respondent burden. The studies with the weakest dietary assessment methods were those using only shopping receipts (two studies), a screening Food Frequency Questionnaire (three studies) or a one- or two-item food group question (five studies). Most studies did not adequately describe their dietary assessment techniques and seven questionnaires were not validated for the target population or nutrient of interest.
Analysis of effectiveness
There was substantial heterogeneity in the estimates of effect of e-learning interventions on many reported outcomes; however, there was relatively little evidence for heterogeneity in the estimates of effect on total fat intake per day, total energy intake per day and BMI. When studies reporting the same outcomes were pooled in a random effects meta-analysis, e-learning interventions were associated with a WMD of 0.24 servings (95% CI 0.04 servings to 0.44 servings; p = 0.019) of fruit and vegetables per day (heterogeneity p < 0.001 and I2 = 83%); WMD of –0.78 g (95% CI –2.5 g to 0.95 g) of total fat consumed (heterogeneity p = 0.18 and I2 = 28%); WMD –0.24 g (95% CI –1.44 g to 0.96 g; p = 0.7) of saturated fat intake (heterogeneity p = 0.001 and I2 = 78%); WMD of –1.4% (95% CI –2.5% to –0.3%; p = 0.012) of total energy consumed from fat (heterogeneity p < 0.001 and I2 = 77%); WMD of 1.45 g (95% CI –0.02 g to 2.92 g; p = 0.053) of dietary fibre per day (heterogeneity p = 0.11 and I2 = 60%); WMD of 4 kcal (95% CI –85 kcal to 93 kcal; p = 0.93) of daily energy intake (heterogeneity p = 0.33 and I2 = 13%); WMD of –0.1 kg/m2 (95% CI –0.7 kg/m2 to 0.4 kg/m2; p = 0.69) in BMI (heterogeneity p = 0.92 and I2 = 0%); and WMD of 0.6 kg (95% CI –3.5 kg to 4.6 kg; p = 0.78) of weight (heterogeneity p < 0.001 and I2 = 86%).
The literature review did not reveal any published economic evaluations of e-learning devices that were based purely on imparting nutritional advice. One published UK study that evaluated a device in conjunction with physical exercise suggested that it was not cost-effective compared with standard methods of providing advice. The base-case results from the E-LEEM model suggested that the incremental cost-effectiveness ratio was approximately £60,000 per QALY. The results were generally robust to most alternative assumptions, except the initial fixed cost of the device. In the base case this cost was assumed to be £854 per person. When the cost was assumed to be £0, the e-learning device was less costly and more effective than the alternative. Expected value of perfect information (EVPI) analysis showed that although the individual-level EVPI was arguably negligible, the population-level value was between £37M and £170M at a willingness to pay of £20,000–30,000 per additional QALY.
Explanations of effectiveness
We analysed the data with a view to determining reasons for the observed variability in effectiveness of e-learning interventions. We considered that potential reasons could include differences in:
- target populations
- target behaviours
- intervention content
- theoretical base
- mode of delivery
- ‘dose’ of intervention
- study quality.
Interventions trialled were so heterogeneous that no firm conclusions could be reached. More mechanistic research is required to address these questions, including whether or not adaptive e-learning tools are better suited for disease management than for health promotion; understanding the psychological mechanisms of action of interventions; and whether or not there is a relationship between ‘dose’ of the intervention and effectiveness. Our statistical assessment of the evidence suggests that trials with lower methodological quality may overestimate effects.
Can e-learning interventions change dietary behaviour?
There are many factors affecting what foods people eat and why, and an intervention targeted at individual behaviour change can address only a selection of these. E-learning has not yet proved itself to be more effective or cost-effective than other behaviour change approaches at an individual level for improving diet, or for preventing or reducing overweight or obesity. Nor is there any research comparing e-learning approaches with population-level approaches to tackling dietary change or reducing obesity.
Are e-learning interventions cost-effective to the NHS?
The results broadly suggest that the e-learning devices are not cost-effective at conventional levels of £20,000–30,000 per QALY gained and the probability that they are cost-effective was 25% at these threshold levels, a result that is perhaps not unsurprising given the relatively modest effects derived from the systematic review of effectiveness. However, the results were sensitive to the assumptions regarding the initial fixed cost of the devices; lowering this value dramatically increased the cost-effectiveness of the devices.
What is the potential population health impact of e-learning interventions?
Fruit and vegetables
Dietary recommendations suggest five servings of fruit and vegetables per person per day; currently adults in the UK (aged 19–65 years) are eating on average 4.4 servings, so an increase of a quarter of a serving would still not raise the average intake to meet the guidelines.
Dietary recommendations suggest that 35% of calories consumed should come from total fat, and 11% of calories from saturated fats. Currently, UK adults consume an average of 34–36% of energy from total fat and 12.8% of energy from saturated fat, so a reduction of 1% would not facilitate achievement of guideline targets.
Recommendations suggest an intake of 18 g of dietary fibre per day, with current intake in UK adults at around 14 g. An increase of 1.5 g would therefore not facilitate achievement of the guideline.
Dietary behaviours are likely to be heavily influenced by macro factors at the environmental, organisational, population and sociocultural levels. These wider determinants of dietary behaviour are unlikely to be changed by individually targeted interventions such as e-learning; the results of this review seem to reflect this, with little evidence of effect found.
Implications for health care
The current clinical and economic evidence base suggests that e-learning devices designed to promote dietary behaviour change will not produce clinically significant changes in dietary behaviour and are at least as expensive as other individual behaviour change interventions.
Implications for research
Although the return on investment (in terms of the expected value of the reduction in decision uncertainty) from further clinical trials of individual e-learning interventions is expected to be high, we believe that further such clinical trials should not be undertaken until theoretically informed work, which addresses the question of which characteristics of the target population, target behaviour, content and delivery of the intervention are likely to lead to positive results, is completed.
The National Institute for Health Research Health Technology Assessment programme.
Harris J, Felix L, Miners A, Murray E, Michie S, Ferguson E, et al. Adaptive e-learning to improve dietary behaviour: a systematic review and cost-effectiveness analysis.. Health Technol Assess 2011;15(37).
NIHR Health Technology Assessment programme
The Health Technology Assessment (HTA) programme, part of the National Institute for Health Research (NIHR), was set up in 1993. It produces high-quality research information on the effectiveness, costs and broader impact of health technologies for those who use, manage and provide care in the NHS. ‘Health technologies’ are broadly defined as all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care.
The research findings from the HTA programme directly influence decision-making bodies such as the National Institute for Health and Clinical Excellence (NICE) and the National Screening Committee (NSC). HTA findings also help to improve the quality of clinical practice in the NHS indirectly in that they form a key component of the ‘National Knowledge Service’.
The HTA programme is needs led in that it fills gaps in the evidence needed by the NHS. There are three routes to the start of projects.
First is the commissioned route. Suggestions for research are actively sought from people working in the NHS, from the public and consumer groups and from professional bodies such as royal colleges and NHS trusts. These suggestions are carefully prioritised by panels of independent experts (including NHS service users). The HTA programme then commissions the research by competitive tender.
Second, the HTA programme provides grants for clinical trials for researchers who identify research questions. These are assessed for importance to patients and the NHS, and scientific rigour.
Third, through its Technology Assessment Report (TAR) call-off contract, the HTA programme commissions bespoke reports, principally for NICE, but also for other policy-makers. TARs bring together evidence on the value of specific technologies.
Some HTA research projects, including TARs, may take only months, others need several years. They can cost from as little as £40,000 to over £1 million, and may involve synthesising existing evidence, undertaking a trial, or other research collecting new data to answer a research problem.
The final reports from HTA projects are peer reviewed by a number of independent expert referees before publication in the widely read journal series Health Technology Assessment.
Criteria for inclusion in the HTA journal series
Reports are published in the HTA journal series if (1) they have resulted from work for the HTA programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.
Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.
The research reported in this issue of the journal was commissioned by the HTA programme as project number 08/57/02. The contractual start date was in April 2009. The draft report began editorial review in October 2010 and was accepted for publication in January 2011. As the funder, by devising a commissioning brief, the HTA programme specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.
The views expressed in this publication are those of the authors and not necessarily those of the HTA programme or the Department of Health.
Editor-in-Chief: Professor Tom Walley CBE
Series Editors: Dr Martin Ashton-Key, Professor Aileen Clarke, Dr Tom Marshall, Professor John Powell, Dr Rob Riemsma and Professor Ken Stein
Associate Editor: Dr Peter Davidson
© 2011 Crown Copyright Top ^