BACKGROUND: Clinical decisions are made based on Cochrane reviews, but the implementation of results of evidence syntheses such as Cochrane reviews is problematic if the evidence is not prepared consistently. All systematic reviews should assess the risk of bias (RoB) in included studies, and in Cochrane reviews, this is done by using Cochrane RoB tool. However, the tool is not necessarily applied according to the instructions. In this study, we aimed to determine the types of bias and their corresponding judgements noted in the 'other bias' domain of Cochrane RoB tool. METHODS: We analyzed Cochrane reviews that included randomized controlled trials (RCTs) and extracted data regarding 'other bias' from the RoB table and accompanying support for the judgment. We categorized different types of other bias. RESULTS: We analyzed 768 Cochrane reviews that included 11,369 RCTs. There were 602 (78%) Cochrane reviews that had 'other bias' domain in the RoB tool, and they included a total of 7811 RCTs. In the RoB table of 337 Cochrane reviews for at least one of the included trials it was indicated that no other bias was found and supporting explanations were inconsistently judged as low, unclear or high RoB. In the 524 Cochrane reviews that described various sources of other bias, there were 5762 individual types of explanations which we categorized into 31 groups. The judgments of the same supporting explanations were highly inconsistent. We found numerous other inconsistencies in reporting of sources of other bias in Cochrane reviews. CONCLUSION: Cochrane authors mention a wide range of sources of other bias in the RoB tool and they inconsistently judge the same supporting explanations. Inconsistency in appraising risk of other bias hinders reliability and comparability of Cochrane systematic reviews. Discrepant and erroneous judgments of bias in evidence synthesis may hinder implementation of evidence in routine clinical practice and reduce confidence in otherwise trustworthy sources of information. These results can help authors of Cochrane and non-Cochrane reviews to gain insight into various sources of other bias that can be found in trials, and also to help them avoid mistakes that were recognized in published Cochrane reviews.

Catholic University of Croatia Zagreb Croatia

Faculty of Health Sciences University of Beira Interior Covilhã Portugal

Faculty of Medicine and Dentistry Palacký University Olomouc Olomouc Czech Republic

Institute of Emergency Medicine in Split Dalmatia County Split Croatia

Medical student University of Split School of Medicine Split Croatia

Medical University of Plovdiv Plovdiv Bulgaria

See more in PubMed

Gluud LL. Bias in clinical intervention research. Am J Epidemiol. 2006;163(6):493–501. doi: 10.1093/aje/kwj069. PubMed DOI

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. doi: 10.1136/bmj.d5928. PubMed DOI PMC

Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]: The Cochrane Collaboration; 2011. Available from https://training.cochrane.org/handbook. Accessed 2 Apr 2019.

Roberts C, Torgerson DJ. Understanding controlled trials - baseline imbalance in randomised controlled trials. Br Med J. 1999;319(7203):185. doi: 10.1136/bmj.319.7203.185. PubMed DOI PMC

Fu R, Vandermeer BW, Shamliyan TA, O'Neil ME, Yazdi F, Fox SH. AHRQ methods for effective health care: handling continuous outcomes in quantitative synthesis. Methods guide for effectiveness and comparative effectiveness reviews. Rockville: Agency for Healthcare Research and Quality (US); 2008. PubMed

Schulz KF. Subverting randomization in controlled trials. JAMA. 1995;274(18):1456–1458. doi: 10.1001/jama.1995.03530180050029. PubMed DOI

A revised tool to assess risk of bias in randomized trials (RoB 2.0). Available at: https://sites.google.com/site/riskofbiastool//welcome/rob-2-0-tool. Accessed 2 Apr 2019.

Higgins J. Chapter 8: assessing risk of bias in included studies. In: Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 510 (updated March 2011): The Cochrane Collaboration; 2011. [Available from https://training.cochrane.org/handbook]. Accessed 2 Apr 2019.

Lundh A, Lexchin J, Mintzes B, Schroll JB, Bero L. Industry sponsorship and research outcome. Cochrane Database Syst Rev. 2017;2:MR000033. PubMed PMC

Bero LA. Why the cochrane risk of bias tool should include funding source as a standard item. Cochrane Database Syst Rev. 2013;12:ED000075. PubMed PMC

Sterne JAC. Why the cochrane risk of bias tool should not include funding source as a standard item. Cochrane Database Syst Rev. 2013;12:ED000076. PubMed PMC

Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMC Med. 2010;8:18. doi: 10.1186/1741-7015-8-18. PubMed DOI PMC

Propadalo Ivana, Tranfic Mia, Vuka Ivana, Barcot Ognjen, Pericic Tina Poklepovic, Puljak Livia. In Cochrane reviews, risk of bias assessments for allocation concealment were frequently not in line with Cochrane's Handbook guidance. Journal of Clinical Epidemiology. 2019;106:10–17. doi: 10.1016/j.jclinepi.2018.10.002. PubMed DOI

Babic A, Tokalic R, Silva Cunha JA, Novak I, Suto J, Vidak M, Miosic I, Vuka I, Poklepovic Pericic T, Puljak L. Risk of bias in cochrane systematic reviews: assessments of risk related to attrition bias are highly inconsistent. bioRxiv. 2018:366658. 10.1101/366658.

Barcot O, Boric M, Poklepovic Pericic T, Cavar M, Dosenovic S, Vuka I, Puljak L. Judgments of risk of bias associated with random sequence generation in trials included in Cochrane systematic reviews are frequently erroneous. BioRxiv. 2018:366674. 10.1101/366674.

Jordan VM, Lensen SF, Farquhar CM. There were large discrepancies in risk of bias tool judgments when a randomized controlled trial appeared in more than one systematic review. J Clin Epidemiol. 2017;81:72–76. doi: 10.1016/j.jclinepi.2016.08.012. PubMed DOI