Health Technology Assessment 1999; Vol. 3: No. 12
monograph in Adobe Acrobat format (538 kbytes)
Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses
1Thomas C Chalmers Centre for Systematic Reviews, Children's
Hospital of Eastern Ontario Research Institute, Canada
Five projects were carried out to achieve these objectives.
A comprehensive list of studies was provided by an electronic search of databases including MEDLINE, EMBASE and the Cochrane Database of Systematic Reviews (CDSR).
Meta-analyses were selected. The inclusion criterion was that the study combined (pooled) the overall results of RCTs included in the meta-analysis.
Data extraction forms were used to extract the necessary data from the articles. Data extraction was completed in duplicate to reduce the chances of error. Inter-rater reliability was calculated before data extraction began.
Quantitative analysis was difficult because of the nature of the research questions and was conducted only for the study examining the effect of RCT quality on estimates of intervention effectiveness. The data for the searching study and the survey, and the descriptive data for the quality assessment study, are discussed mostly in a qualitative manner.
The overlap of articles and journals between MEDLINE and EMBASE was 80% and 87%, respectively. The database of 491 articles that was used comprised 455 meta-analyses identified by the MEDLINE search and 36 meta-analyses in the CDSR.
Response rates from the survey were 78%, 74% and 59% for reviewers (n = 121), methodologists (n = 55) and editors (n = 63), respectively. Over 90% of respondents stated that assessment and reporting of quality of RCTs included in meta-analyses was very or somewhat important. The use of RCT design features as inclusion criteria and using quality assessments to conduct sensitivity analyses were the most frequently endorsed methods of incorporating the quality assessments into meta-analyses. Most respondents believed that guidelines on the assessment and reporting of the quality of randomised trials would increase the rigour of reporting of published meta-analyses and make interpretation easier.
Of a sample of 240 meta-analyses, trial quality was assessed in 48% and in half of these data on the reproducibility of the assessments were provided. Of the meta-analyses that assessed quality, only 25% incorporated trial quality into the analyses.
Masked and unmasked quality assessments were carried out on 127 RCTs included in 11 meta-analyses in the database. The assessments were made using a validated scale (1-5, higher scores indicate superior reporting) and individual components known to affect estimates of intervention effectiveness. Masked quality assessment provided significantly higher scores (mean = 2.74; standard deviation (SD) = 1.10) than unmasked assessments (mean = 2.55; SD = 1.20). Low-quality trials were associated with an increase of 34% in estimate of benefit (ratio of odds ratios (ROR) = 0.66; 95% confidence interval (CI): 0.52, 0.83) compared with high-quality trials. Trials using inadequate allocation concealment, compared with those using adequate methods, were also associated with an increased estimate of benefit of 37% (ROR = 0.63; 95% CI: 0.45, 0.88). The average treatment benefit across all trials was 39% (OR = 0.61; 95% CI: 0.57, 0.65). Including only trials with low quality scores increased this effect to 52% (OR = 0.48; 95% CI: 0.43, 0.54), whereas including only trials with high quality scores reduced the effect to 29% (OR = 0.71; 95% CI: 0.65, 0.77). Using all the trial scores as quality weights reduced the effect to 35% (OR = 0.65; 95% CI: 0.59, 0.71) and resulted in the least statistical heterogeneity.
Indexing inconsistencies within and across databases pose challenges in searching for systematic reviews or meta-analyses. Our results suggest that it is necessary to search multiple databases to identify all relevant information. Journal indexers, authors and editors should collaborate to develop and implement criteria to help users of systematic reviews and meta-analyses identify relevant publications.
The systematic reviewers, methodologists and journal editors surveyed believed that assessment of trial quality was important. This contrasts with the infrequent reporting of trial quality in published meta-analyses. Future studies should address the issue of quality assessment. Consistent reporting of the design features of RCTs may help to enhance the rigour and clinical interpretability of meta-analyses.
Among a sample of meta-analyses from the database, individual components and scales were the methods most commonly used to assess trial quality. However when quality assessments were made, in most cases they were not incorporated into the analysis. This is important because the incorporation of quality assessments can alter the estimate of the benefit of intervention, regardless of which method of assessment is used.
The results from these studies also suggest that certain characteristics of the design and execution of RCTs impact on the probability of bias, and further research is needed on this. Investigations are also needed to clarify the value of masking studies before quality assessment and to determine the advantages of the various approaches to incorporate quality assessments into the analyses. Until such empirical evidence is presented, the guidelines outlined below are a useful tool with which meta-analysts, editors, peer reviewers and readers can deal with issues pertaining to quality assessment of randomised trials included in a meta-analysis.
Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, Jones A, et al. Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. Health Technol Assessment 1999; 3(12).
NHS R&D HTA ProgrammeThe overall aim of the NHS R&D Health Technology Assessment (HTA) programme is to ensure that high quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and work in the NHS. Research is undertaken in those areas where the evidence will lead to the greatest benefits to patients, either through improved patient outcomes or the most efficient use of NHS resources. The Standing Group on Health Technology advises on national priorities for health technology assessment. Six advisory panels assist the Standing Group in identifying and prioritising projects. These priorities are then considered by the HTA Commissioning Board supported by the National Coordinating Centre for HTA. This report is one of a series covering acute care, diagnostics and imaging, methodology, pharmaceuticals, population screening, and primary and community care. It was identified as a priority by the Methodology Panel and funded as project number 93/52/04. The views expressed in this publication are those of the authors and not necessarily those of the Standing Group, the Commissioning Board, the Panel members or the Department of Health. The editors wish to emphasize that funding and publication of this research by the NHS should not be taken as implicit support for the recommendations for policy contained herein. In particular, policy options in the area of screening will be considered by the National Screening Committee. This Committee, chaired by the Chief Medical Officer, will take into account the views expressed here, further available evidence and other relevant considerations. Reviews in Health Technology Assessment are termed 'systematic' when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.
©1999 Crown Copyright