View/Download full
monograph in Adobe Acrobat format (675 kbytes)
View/Download this
4-page summary in Adobe Acrobat format (suitable for printing) Health Technology Assessment 2000; Vol. 4: No. 10
Executive summary
View/Download full
monograph in Adobe Acrobat format (675 kbytes)
View/Download this
4-page summary in Adobe Acrobat format (suitable for printing)
F Song1*
AJ Eastwood1
S Gilbody1
L Duley2
AJ Sutton3
1 NHS Centre for Reviews and Dissemination, University of York, York, UK
2 Magpie Trial, Institute of Health Sciences, Oxford, UK
3 Department of Epidemiology and Public Health, University of Leicester, Leicester, UK
* Corresponding author
Literature review is becoming increasingly important in summarising research evidence for clinical and health policy decision making because of a rapidly expanding volume of medical research. However, the results of literature reviews will be misleading if the published studies comprise a biased sample of all the studies that have been conducted.
The term “dissemination profile” could be used to describe the accessibility of research results or the possibility of research findings being identified by potential users. The spectrum of the dissemination profile ranges from completely inaccessible to easily accessible, according to whether, when, where and how research is published.
This review aimed to identify systematically and appraise studies that have examined methodological issues and provided empirical evidence about publication bias and other dissemination-related biases, including biases due to the time, type and language of publication, multiple publication, selective citation of references, database index bias, and biased media attention. The review sought to answer the following questions:
This report includes a systematic review of publication and related biases, and a survey of publication bias in published systematic reviews.
The following databases were searched to identify relevant literature concerning empirical evidence and methodological issues pertaining to publication and related biases: the Cochrane Review Methodology Database, MEDLINE, EMBASE, BIDS, Library and Information Science Abstracts, PsycINFO, Sociofile, ERIC, Dissertation Abstracts, MathSci, British Education Index, SIGLE and ASSIA. The reference lists of the identified articles were also checked.
The results of searches of electronic databases were checked independently by two reviewers and any disagreements discussed. Full publications for studies that were considered to be potentially relevant were obtained and their suitability for inclusion independently assessed by at least two reviewers. All studies relevant to publication and related biases were included, except if the issue of publication bias was not a major topic. Data from included studies were collected by one reviewer by using a data-extraction form and then checked by another reviewer.
A sample of 193 systematic reviews was taken from the Database of Abstracts of Reviews of Effectiveness (NHS Centre for Reviews and Dissemination at the University of York) to identify further evidence of publication and related biases and to illustrate the methods used for dealing with publication bias. These reviews were assessed independently by two reviewers using a data-extraction form.
The empirical evidence demonstrates that studies with significant results or favourable results are more likely to be published or cited than those with non-significant or unfavourable results. Studies with significant results are often published earlier than those with non-significant results. Limited and often indirect evidence indicates only the possibility of full publication bias, outcome reporting bias, duplicate publication bias, language bias and database bias. There is some evidence concerning the existence of citation bias and media attention bias.
The important consequences of publication bias include the avoidable suffering of patients and the waste of limited resources. However, there is little empirical evidence relating to the impact of publication and related biases on health policy, clinical decision making and the outcome of patient management.
Investigators, peer reviewers, editors and funding bodies may all be responsible for the existence of publication bias. Some evidence suggests that authors or investigators may be the main source of this bias, for not writing up or not submitting studies with null or unimportant results. However, it should be recognised that the decision to write up an article and then submit it may be affected by pressure from research sponsors and instruction from journal editors. Evidence shows that the interest of research sponsors can restrict the dissemination of research findings. The large potential variation in results obtained across similar studies that can easily be conducted and abandoned will further exacerbate the biased selection of findings for publication.
Because of their space limitations and need to maintain newsworthiness, it is unlikely that conventional paper journals can solve the problem of the selective publication of studies that produce striking results. For the purpose of reducing publication bias, peer-reviewed electronic journals that are without limitations of space are required. More importantly, editorial policy needs to be changed to accept for publication clinical trials that are based on methodological criteria only and not on the impact of their findings.
Clearly, the ideal solution to publication bias is the prospective, universal registration of all studies at their inception. Although the registration of all studies cannot be realised in the near future, there are many encouraging signs that there will be more registries established as a result of initiatives from government or industry. Large-scale confirmatory studies may be an alternative in the prevention of the consequences of publication bias.
The methods available for dealing with publication and related biases in systematic reviews include literature searching, locating unpublished studies, assessment of the risk of publication and related biases, several methods for detecting publication bias in meta-analyses, and updating systematic reviews. The statistical methods are by nature indirect and exploratory, and often based on certain strict assumptions that can be difficult to justify in the real world. The attempt at identifying or adjusting for publication bias in a systematic review should mainly be used for the purpose of sensitivity analysis.
This survey indicates that literature searching was clearly inadequate in some published systematic reviews. Potential publication bias was ignored and the available methods for dealing with such bias were not used in most of these reviews. When they are used to estimate possible publication bias at the stage of literature review, the available methods were far from adequate and their usefulness was strictly limited. The problem of publication and related biases was dealt with more often in reviews containing a meta-analysis than in the narrative systematic reviews.
Although the extent, direction and impact of publication and related biases are uncertain and may vary greatly depending on circumstances, it seems reasonable to conclude that studies with significant or favourable results are more widely disseminated than those with non-significant or unfavourable results. The potential problem of publication and related biases should be taken into consideration in the field of health technology assessment. All funded or approved studies should be prospectively registered. The risk of publication bias should be assessed in all systematic reviews.
Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. Publication and related biases. Health Technol Assess 2000;4(10).
The overall aim of the NHS R&D Health Technology Assessment (HTA) programme is to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and work in the NHS. Research is undertaken in those areas where the evidence will lead to the greatest benefits to patients, either through improved patient outcomes or the most efficient use of NHS resources.
The Standing Group on Health Technology advises on national priorities for health technology assessment. Six advisory panels assist the Standing Group in identifying and prioritising projects. These priorities are then considered by the HTA Commissioning Board supported by the National Coordinating Centre for HTA (NCCHTA).
This report is one of a series covering acute care, diagnostics and imaging, methodology, pharmaceuticals, population screening, and primary and community care. It was identified as a priority by the Methodology Group and funded as project number 95/12/01.
The views expressed in this publication are those of the authors and not necessarily those of the Standing Group, the Commissioning Board, the Panel members or the Department of Health. The editors wish to emphasize that funding and publication of this research by the NHS should not be taken as implicit support for the recommendations for policy contained herein. In particular, policy options in the area of screening will be considered by the National Screening Committee. This Committee, chaired by the Chief Medical Officer, will take into account the views expressed here, further available evidence and other relevant considerations.
Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.
Criteria for inclusion in the HTA monograph series
Reports are published in the HTA monograph series if (1) they have resulted from work either prioritised by the Standing Group on Health Technology, or otherwise commissioned for the HTA programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.
Series Editors: Andrew Stevens, Ruairidh Milne, Ken Stein and John Gabbay
Monograph Editorial Manager: Melanie Corris
The editors have tried to ensure the accuracy of this report but cannot accept responsibility for any errors or omissions. They would like to thank the referees for their constructive comments on the draft document.
©2000 Crown Copyright