Health Technology Assessment 2003; Vol 7: number 41
Executive SummaryView/Download full monograph in Adobe Acrobat format (528 kbytes)
View/Download 4-page summary in Adobe Acrobat format (suitable for printing)
of Pediatrics, Faculty of Medicine, University of Ottawa, Canada
2Department of Epidemiology and Community Medicine, Faculty of Medicine, University of Ottawa, Canada
3Chalmers Research Group, Childrens Hospital of Eastern Ontario Research Institute, Ottawa, Canada
4BioMedical Data Sciences, GlaxoSmithKline, Toronto, Canada
5Department of Pediatrics, University of Alberta, Edmonton, Canada
* Corresponding author
In an era of evidence-based healthcare, systematic reviews are becoming increasingly important as a source of evidence for decision-making. They afford the reader an opportunity to review quickly the totality of evidence regarding a particular intervention. Ideally, the systematic review process of randomised controlled trials (RCTs) provides the reader with a bias-free estimate of the effects of the intervention under consideration.
There is now evidence regarding the influence that several factors in the review process have on the results of a systematic review. For example, excluding unpublished studies, compared with their inclusion, can exaggerate the estimates of an interventions effectiveness by 15%, on average.
The role of including reports of RCTs reported in languages other than English (LOE) (i.e. language restriction) remains uncertain. Such studies are difficult to identify and retrieve. The costs of including these studies can be prohibitive for the average reviewer. Yet excluding them from the systematic review process might introduce substantial bias, make the review process flawed and exaggerate the results of the review. Confounding the decision to exclude these studies is whether they are investigating a conventional medicinal (CM) intervention, such as methylphenidate for attention deficit hyperactivity disorder, or a complementary and alternative medicine (CAM) intervention, such as hypnosis for treating migraines. Traditionally, CAM has been investigated in countries whose first language is not English.
We set out to assemble a large dataset of language restricted and language inclusive systematic reviews, including both CM and CAM interventions. We also assessed the quality of these different types of systematic reviews and their associated RCTs and compared the quality of systematic reviews investigating a CM intervention with those reviews examining CAM interventions. We also examined whether language restrictions compared with language inclusions exaggerate the estimates of an interventions effectiveness. Finally we evaluated whether language restrictions of conventional interventions are similar to those for CAM interventions, and whether these results are influenced by other issues, including statistical heterogeneity and publication bias, in the systematic review process.
A systematic review was included if the primary data sources were reports of RCTs identified through a collection of systematic reviews assembled by the Chalmers Research Group. This collection was based on searching Medline, Embase and the Cochrane Database of Systematic Reviews. We added to this collection for the present work by searching Embase and the Centralised Information Service for Complementary Medicine.
The methodology section of the systematic review had to state explicitly whether the search was limited to identifying and including English RCTs only, or whether RCTs of all languages were eligible for inclusion. Systematic reviews were excluded if there was no mention of language restriction or inclusion.
Three types of systematic reviews were included: language restricted systematic reviews, meaning that no reports of RCTs reported in LOE were included in the quantitative data synthesis (i.e. language restricted systematic reviews); language inclusive/English language (EL) systematic reviews that searched for reports of RCTs in LOE but did not find any and, hence, could not include any, in the quantitative data synthesis; and systematic reviews that searched for reports of RCTs in LOE and included them in the quantitative data synthesis (i.e. language inclusive/LOE systematic reviews).
We estimated that 45 language restricted and 45 language inclusive systematic reviews would be required to detect a 25% difference in the ratio of odds ratios of intervention (ROR) between trials published in a an LOE and reports of English language RCTs, on a logarithmic scale.
We assessed the quality of reports of all three types of systematic reviews and the RCTs contained in the language inclusive/LOE reviews. All the assessments were completed using state of the art assessment instruments.
Characteristics (e.g. assessment of publication bias) of each systematic review were abstracted using a standardised data collection form.
Similarly, for each included RCT, we extracted the first authors name, journal, year of publication, language of publication and whether or not it related to CAM. For the primary outcome, we also extracted the number of events and patients in the control group and the number of events and patients in the experimental group.
We applied Fishers exact test to compare the three different types of systematic reviews with respect to their reporting characteristics and the systematic review quality assessment tool. We computed the log ROR of LOE trials versus EL trials for each systematic review and pooled this information across systematic reviews to examine the influence that language of publication and type of intervention (CM, CAM) have on the estimates of intervention effect. Several sensitivity analyses were performed.
We included 130 systematic reviews: 50 language restricted, 32 language inclusive/EL and 48 language inclusive/LOE systematic reviews. Approximately 20% of the reviews were investigating CAM. The language inclusive/LOE reviews included the largest number of RCTs and participants. The LOE RCTs were published predominantly in French and German.
Language inclusive/LOE systematic reviews were of the highest quality compared with the other types of reviews, scoring 57% of the maximum possible score. The CAM reviews were of higher quality, averaging 71% of their maximum possible score, compared with the CM reviews. There were only minor differences in the quality of reports of RCTs published in English compared with the eight other languages included in this analysis. However, there are inconsistent differences in the quality of LOE reports depending upon the type of intervention.
The present results, and those reported previously, suggest that excluding reports of RCTs in LOE from the analytical part of a systematic review is a reasonable way to conduct a review [random effects model (RE) ROR = 1.02; 95% confidence interval (CI): = 0.83 to 1.26]. Because the present research and previous efforts have not included every type of CM RCT and the resulting possibility of the uncertainty as to when bias will be present by excluding LOE, it is always prudent to perform a comprehensive search for all evidence. This result only applies to reviews investigating the benefits of CM interventions. This does not imply that systematic reviewers should neglect reports in LOE. We recommend that systematic reviewers search for reports regardless of the language of their publication. There may be merit in including them in some aspects of the review process although this decision is likely to depend on several factors, including fiscal and other resources being available.
However, language restrictions significantly shift the estimates of an interventions effectiveness when the intervention is CAM. Here, excluding trials reported in LOE, compared with their inclusion, resulted in a reduced intervention effect, 63% on average (RE ROR = 1.63; 95% CI: 1.03 to 2.60).
The present results do not appear to be influenced by statistical heterogeneity and publication bias.
With the exception of CAM systematic reviews, the quality of recently published systematic reviews is less than optimal. Language inclusive/LOE systematic reviews appear to be a marker for a better quality systematic review. Language restrictions do not appear to bias the estimates of a conventional interventions effectiveness. However, there is substantial bias in the results of a CAM systematic review if LOE reports are excluded from it.
Consideration of the development of a national database of systematic reviews is likely to facilitate meta-epidemiology research undertaken in the UK and elsewhere.
The quality of reporting of systematic reviews of RCTs needs improvement. This is most likely to be achieved if authors and medical journal editors agree to a standardised and evidence-based way of reporting. The quality of reporting of meta-analysis of randomised trials (QUOROM) statement is one option to consider for systematic reviews. Likewise, the consolidated standards of reporting randomised trials (CONSORT) statement is likely to improve the quality of reporting of randomised trials.
To keep QUOROM and CONSORT up to date, regular meetings of these groups should be encouraged.
A more in-depth examination of CAM trials, particularly those conducted in Asian countries, and their influence on the conduct of systematic reviews is required.
Aspects of CAM methodology and content need to be incorporated in critical appraisal skills training programmes.
Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health Technol Assess 2003;7(41).
The NHS R&D Health Technology Assessment (HTA) Programme was set up in 1993 to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and provide care in the NHS.
Initially, six HTA panels (pharmaceuticals, acute sector, primary and community care, diagnostics and imaging, population screening, methodology) helped to set the research priorities for the HTA Programme. However, during the past few years there have been a number of changes in and around NHS R&D, such as the establishment of the National Institute for Clinical Excellence (NICE) and the creation of three new research programmes: Service Delivery and Organisation (SDO); New and Emerging Applications of Technology (NEAT); and the Methodology Programme.
The research reported in this monograph was identified as a priority by the HTA Programmes Methodology Panel and was funded as project number 96/52/99.
The views expressed in this publication are those of the authors and not necessarily those of the Methodology Programme, HTA Programme or the Department of Health. The editors wish to emphasise that funding and publication of this research by the NHS should not be taken as implicit support for any recommendations made by the authors.
Criteria for inclusion in the HTA monograph series
Reports are published in the HTA monograph series if (1) they have resulted from work commissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.
Reviews in Health Technology Assessment are termed systematic when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.
Methodology Programme Director: Professor Richard Lilford
HTA Programme Director: Professor Kent Woods
Series Editors: Professor Andrew Stevens, Dr Ken Stein, Professor John Gabbay, Dr Ruairidh Milne and Dr Rob Riemsma
Managing Editors: Sally Bailey and Sarah Llewellyn Lloyd
The editors and publisher have tried to ensure the accuracy of this report but do not accept liability for damages or losses arising from material published in this report. They would like to thank the referees for their constructive comments on the draft document.
© 2003 Crown Copyright Top ^