F Song,^1,2* S Parekh,^1,2 L Hooper,¹ YK Loke,¹ J Ryder,¹ AJ Sutton,³ C Hing,⁴ CS Kwok,¹ C Pang,¹ and I Harvey¹

¹School of Medicine, Health Policy and Practice, University of East Anglia, Norwich, UK
²School of Allied Health Professions, University of East Anglia, Norwich, UK
³Department of Health Sciences, University of Leicester, UK
⁴Watford General Hospital, Hertfordshire, UK

*Corresponding author

Background

The validity of research synthesis is threatened if studies with significant or striking findings are more likely to be published than those with non-significant results. A previous Health Technology Assessment (HTA) monograph published in 2000 by the present authors reviewed studies on publication and related biases. Since then, many new studies on publication and related biases have been published. This report aims to update the 2000 HTA monograph on publication bias by synthesising findings from previous studies and newly indentified ones.

Objectives

• To identify and appraise empirical studies on publication and related biases published since 1998.
• To assess the usefulness and limitations of available methods to deal with publication and related biases.
• To examine in a random sample of published systematic reviews, measures taken by the authors to prevent, reduce and detect different types of dissemination bias.

Methods

Part I: Review of evidence and method studies

Study selection

The report included evidence studies that provided empirical evidence on the existence, consequences, causes and/or risk factors of dissemination bias; and method studies that developed or evaluated methods for preventing, reducing or detecting dissemination bias.

Data sources

The following electronic databases were searched: Cochrane Methodology Register Database (CMRD), MEDLINE, EMBASE, AMED and CINAHL. The main literature search was conducted in August 2008 and a final search of PubMed, PsycINFO and OpenSIGLE was conducted in May 2009 to identify more recently published studies. We also examined reference lists of retrieved studies.

Data extraction and synthesis

The identified studies were classified by one reviewer as evidence or method studies and checked by a second reviewer. One reviewer extracted data directly into tables (specifically designed according to types of bias or methods), which were checked by a second reviewer. Evidence from empirical studies was summarised narratively. Where appropriate, the results have been quantitatively pooled.

Part II: Survey of published systematic reviews

We searched MEDLINE for systematic reviews published in 2006, and randomly selected 100 reviews of effects of health-care interventions, 50 reviews of diagnostic accuracy, 100 reviews of association between risk factors and health outcomes, and 50 reviews of gene-disease associations. We assessed the methods used to deal with publication and related biases in these systematic reviews.

Results

Empirical evidence on dissemination bias

Updated analyses of data from cohort studies confirmed findings from the previous HTA report that studies with significant or positive results are more likely to be published than those with non-significant or negative results. Publication bias occurs mainly before the presentation of findings at conferences and before the submission of manuscripts to journals. Recent high-quality studies have provided convincing evidence that outcome reporting bias exists and has an important impact on the pooled summary in systematic reviews. Studies with significant results tend on average to be published earlier than studies with non-significant results, although the new evidence is less clear than that from the previous review. New empirical evidence suggests that published studies tend to report a greater treatment effect than those from grey literature. However, for individual cases, the direction of bias is unpredictable, and grey literature studies may be relatively small and of poor quality. The impact of non-English language studies was highly heterogeneous. Exclusion of non-English language studies appears to result in a particularly high risk of bias in some areas of research such as complementary and alternative medicine. The updated review also identified limited evidence on citation bias, duplicate publication bias, place of publication bias, database or index bias, country bias and media attention bias.

Limitations of the available evidence

Empirical studies on publication and related biases have focused mainly on certain areas of research such as clinical trials of health-care interventions. When studies are classified as positive or important, bias may be introduced due to inevitable subjectivity. Much of the empirical evidence comes from case reports, which may be selectively reported because of their striking findings. Cohort studies often included studies that were diverse in terms of design and research questions. It is usually impossible to exclude the impact of confounding factors on the observed association between study results and publication status.

Consequences of research dissemination bias

The most important consequences of publication bias include avoidable suffering of patients and waste of limited resources. This updated review identified only a few new cases that indicate the detrimental impact of publication and related biases. Consequences of publication and related biases are different for different types of research studies. Dissemination bias can jeopardise the integrity of scientific research.

Sources of publication bias

The dissemination profile of a research finding is determined by the interests of research sponsors, investigators, peer-reviewers and editors. The updated review identified further evidence indicating that publication bias is often due to investigators’ failure to write up and submit, although it should be recognised that the investigators’ decision to write up an article and then submit it may be affected by pressure from research sponsors, preferences of journal editors, and the requirements of the research award system. Newly identified and previous included evidence suggested that the interests of research sponsors, particularly industry’s commercial interests, can restrict the dissemination of the research findings. Studies that can be conducted without the use of large amounts of resource investment, and those that are of great variations in results are more subject to publication bias.

Methods to prevent, reduce or detect publication and related bias

The available methods can be classified according to the stage of a literature review: to prevent publication bias before a literature review (e.g. prospective registration of trials), to detect publication bias during a literature review (e.g. locating unpublished studies, funnel plot and related tests, sensitivity analysis modelling), or to minimise the impact of publication bias after a literature review (e.g. confirmatory large-scale trials, updating the systematic review).

The first step for the prevention of publication bias is a wide public awareness of detrimental consequences of publication bias, and the need for the results of all studies to be made accessible. One important solution to publication bias is the prospective registration of all studies at inception. The compulsory policy of trial registration adopted by the International Committee of Medical Journals in 2004 may be the most influential initiative so far to promote prospective registration of clinical trials. The World Health Organization (WHO) initiated a project in 2005 to set international standards for clinical trial registration. Further action through government regulations (e.g. the FDA Modernisation Act in the USA) may still be required. In spite of the greater risk of publication bias, the prospective registration and disclosure of data from unpublished basic research, observational studies and early stage exploratory trials has faced considerable difficulties.

The development of prospective trial registration itself is not sufficient for the prevention of publication bias. It is important to make sure that results of registered trials are publicly accessible. The usefulness of trial registrations relies on systematic reviewers searching them, using the data they provide and spending time contacting trialists where studies have not yet been published.

The recent development of clinical trial registration and electronic publication of results from clinical trials will facilitate the identification and location of ongoing or unpublished clinical trials. Funnel plot and related statistical tests have been widely used to assess publication bias. Unfortunately, the interpretation of results of funnel plot tests was often too simplistic and likely misleading. Many sophisticated modelling methods have not been widely used in systematic reviews, possibly because of their complexity and lack of user friendly software.

Survey of published systematic reviews

Compared with systematic reviews published in 1996, recent systematic reviews of health-care interventions are making greater efforts to locate and include non-English language studies (47% versus 30%), and grey literature or unpublished studies (53% versus 35%). There was also an increased use of available methods to test for publication bias in recent reviews (22% versus 17%). Grey literature, unpublished studies or non-English language studies were more likely to be searched for in reviews of treatment efficacy or diagnostic accuracy than in reviews of epidemiological studies. However, the risk of publication bias was less likely to be tested in reviews of treatment and diagnosis as compared with reviews of epidemiological studies.

Conclusions

Dissemination of research findings is likely to be a biased process, although the actual impact of such bias is still uncertain, depending on specific circumstances. Therefore, the potential problem of research dissemination bias should be taken into consideration by all who are involved in evidence-based decision making. The recent initiatives in the prospective registration of clinical trials and the endorsement of reporting guidelines may prevent or reduce publication and reporting bias in future systematic reviews of clinical trials, although prospective registration of basic research, early stage clinical studies and observational studies is still underdeveloped. However, trial registers will only be helpful in reducing publication bias if the results of registered trials are accessible. In systematic reviews, measures can be taken to minimise the impact of research dissemination bias by systematically searching for published and unpublished studies. All statistical methods, simple or complex, are by nature indirect and exploratory, and are often based on certain assumptions that can be difficult to justify. The available statistical methods can be useful for the purpose of sensitivity analyses.

Recommendations for future research

• Further empirical research is needed to evaluate the effect of prospective registration of studies, open access policy and improved publication guidelines in the prevention of research dissemination bias.
• The role of developments in computer science and information technology for the prevention of research dissemination bias needs to be investigated by further research.
• The impact of publication bias on health decision-making and the outcomes of patient management need to be investigated by further research.
• Methods that can be used to assess qualitatively the risk of publication bias in systematic reviews need to be developed by further research.
• Further research should focus on the practical application of the available statistical methods.

NIHR Health Technology Assessment programme

The Health Technology Assessment (HTA) programme, part of the National Institute for Health Research (NIHR), was set up in 1993. It produces high-quality research information on the effectiveness, costs and broader impact of health technologies for those who use, manage and provide care in the NHS. 'Health technologies' are broadly defined as all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care.

The research findings from the HTA programme directly influence decision-making bodies such as the National Institute for Health and Clinical Excellence (NICE) and the National Screening Committee (NSC). HTA findings also help to improve the quality of clinical practice in the NHS indirectly in that they form a key component of the 'National Knowledge Service'.

The HTA programme is needs led in that it fills gaps in the evidence needed by the NHS. There are three routes to the start of projects.

First is the commissioned route. Suggestions for research are actively sought from people working in the NHS, from the public and consumer groups and from professional bodies such as royal colleges and NHS trusts. These suggestions are carefully prioritised by panels of independent experts (including NHS service users). The HTA programme then commissions the research by competitive tender.

Second, the HTA programme provides grants for clinical trials for researchers who identify research questions. These are assessed for importance to patients and the NHS, and scientific rigour.

Third, through its Technology Assessment Report (TAR) call-off contract, the HTA programme commissions bespoke reports, principally for NICE, but also for other policy-makers. TARs bring together evidence on the value of specific technologies.

Some HTA research projects, including TARs, may take only months, others need several years. They can cost from as little as £40,000 to over £1 million, and may involve synthesising existing evidence, undertaking a trial, or other research collecting new data to answer a research problem.

The final reports from HTA projects are peer reviewed by a number of independent expert referees before publication in the widely read journal series Health Technology Assessment.

Criteria for inclusion in the HTA journal series

Reports are published in the HTA journal series if (1) they have resulted from work for the HTA programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed 'systematic' when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

The research reported in this issue of the journal was commissioned by the National Coordinating Centre for Research Methodology (NCCRM), and was formally transferred to the HTA programme in April 2007 under the newly established NIHR Methodology Panel. The HTA programme project number is 06/92/02. The contractual start date was in October 2007. The draft report began editorial review in November 2008 and was accepted for publication in June 2009. The commissioning brief was devised by the NCCRM who specified the research question and study design. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the HTA programme or the Department of Health.

Editor-in-Chief: Professor Tom Walley CBE
Series Editors: Dr Martin Ashton-Key, Dr Aileen Clarke, Professor Chris Hyde, Dr Tom Marshall, Dr John Powell, Dr Rob Riemsma and Professor Ken Stein