Publication selection bias undermines the systematic accumulation of evidence. To assess the extent of this problem, we survey over 68,000 meta-analyses containing over 700,000 effect size estimates from medicine (67,386/597,699), environmental sciences (199/12,707), psychology (605/23,563), and economics (327/91,421). Our results indicate that meta-analyses in economics are the most severely contaminated by publication selection bias, closely followed by meta-analyses in environmental sciences and psychology, whereas meta-analyses in medicine are contaminated the least. After adjusting for publication selection bias, the median probability of the presence of an effect decreased from 99.9% to 29.7% in economics, from 98.9% to 55.7% in psychology, from 99.8% to 70.7% in environmental sciences, and from 38.0% to 29.7% in medicine. The median absolute effect sizes (in terms of standardized mean differences) decreased from d = 0.20 to d = 0.07 in economics, from d = 0.37 to d = 0.26 in psychology, from d = 0.62 to d = 0.43 in environmental sciences, and from d = 0.24 to d = 0.13 in medicine.

Centre for Environmental Sciences Hasselt University Hasselt Belgium

Department of Biomedical Data Science Stanford University School of Medicine Stanford California USA

Department of Economics Deakin University Geelong Victoria Australia

Department of Economics University of Göttingen Göttingen Germany

Department of Epidemiology and Population Health Stanford University School of Medicine Stanford California USA

Department of Experimental Psychology University College London London UK

Department of Methodology London School of Economics and Political Science London UK

Department of Pediatric Neurology UMC Utrecht Brain Center University Medical Center Utrecht Utrecht University Utrecht Netherlands

Department of Psychological Methods University of Amsterdam Amsterdam Netherlands

Department of Psychology University of Amsterdam Amsterdam Netherlands

Department of Statistics Stanford University School of Humanities and Sciences Stanford California USA

Doctoral Centre School of Social Sciences Heriot Watt University Edinburgh UK

Institute of Computer Science Czech Academy of Sciences Prague Czech Republic

Meta Research Innovation Center at Stanford Stanford California USA

School of Psychology University of Sussex Sussex UK

Stanford Prevention Research Center Department of Medicine Stanford University School of Medicine Stanford California USA

Zobrazit více v PubMed

Chavalarias D, Ioannidis JP. Science mapping analysis characterizes 235 biases in biomedical research. J Clin Epidemiol. 2010;63(11):1205‐1215.

Dwan K, Altman DG, Arnaiz JA, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PloS One. 2008;3(8):e3081.

Rosenthal R, Gaito J. Further evidence for the cliff effect in interpretation of levels of significance. Psychol Rep. 1964;15(2):570.

Wicherts JM. The weak spots in contemporary science (and how to fix them). Animals. 2017;7(12):90‐119.

Otte WM, Vinkers CH, Habets PC, IJzendoorn vDG, Tijdink JK. Analysis of 567,758 randomized controlled trials published over 30 years reveals trends in phrases used to discuss results that do not reach statistical significance. PLoS Biol. 2022;20(2):e3001562.

John LK, Loewenstein G, Prelec D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci. 2012;23(5):524‐532.

Fiedler K, Schwarz N. Questionable research practices revisited. Soc Psychol Personal Sci. 2016;7(1):45‐52.

De Winter JC, Dodou D. A surge of p‐values between 0.041 and 0.049 in recent decades (but negative results are increasing rapidly too). PeerJ. 2015;3:e733.

Fanelli D, Costas R, Ioannidis JP. Meta‐assessment of bias in science. Proc Natl Acad Sci. 2017;114(14):3714‐3719.

Ioannidis JP, Stanley T, Doucouliagos H. The power of bias in economics research. Econ J. 2017;127(605):F236‐F265.

Mathur VW. Finding common ground in meta‐analysis “wars” on violent video games. Perspect Psychol Sci. 2019;14(4):705‐708.

Stanley TD, Carter EC, Doucouliagos H. What meta‐analyses reveal about the replicability of psychological research. Psychol Bull. 2018;144(12):1325‐1346.

Van Aert RC, Wicherts JM, Van Assen MA. Publication bias examined in meta‐analyses from psychology and medicine: a meta‐meta‐analysis. PloS One. 2019;14(4):e0215052.

Schwab S, Kreiliger G, Held L. Assessing treatment effects and publication bias across different specialties in medicine: a meta‐epidemiological study. BMJ Open. 2021;11(9):e045942.

Kühberger A, Fritz A, Scherndl T. Publication bias in psychology: a diagnosis based on the correlation between effect size and sample size. PloS One. 2014;9(9):e105825.

Fanelli D. “Positive” results increase down the hierarchy of the sciences. PloS One. 2010;5(4):e10068.

Ioannidis JP. Excess significance bias in the literature on brain volume abnormalities. Arch Gen Psychiatry. 2011;68(8):773‐780.

Scheel AM, Schijen MR, Lakens D. An excess of positive results: comparing the standard psychology literature with registered reports. Adv Methods Pract Psychol Sci. 2021;4(2):1‐12.

Sterling TD. Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. J Am Stat Assoc. 1959;54(285):30‐34.

Fanelli D. Negative results are disappearing from most disciplines and countries. Scientometrics. 2012;90(3):891‐904.

Deressa TK, Stern DI, Vangronsveld J, et al. More than half of statistically significant research findings in the environmental sciences are actually not. Submitted for publication 2022.

Sladekova M, Webb LE, Field AP. Estimating the change in meta‐analytic effect size estimates after the application of publication bias adjustment methods. Psychol Methods. 2022;28:664‐686.

Viechtbauer W. Conducting meta‐analyses in R with the metafor package. J Stat Softw. 2010;36(3):1‐48.

Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to Meta‐Analysis. John Wiley & Sons; 2009.

Maier M, Bartoš F, Wagenmakers EJ. Robust Bayesian meta‐analysis: addressing publication bias with model‐averaging. Psychol Methods. 2022;28:107‐122.

Bartoš F, Maier M, Wagenmakers EJ, Doucouliagos H, Stanley TD. Robust Bayesian meta‐analysis: model‐averaging across complementary publication bias adjustment methods. Research Synthesis Methods. 2022;14(1):99‐116.

Hoeting JA, Madigan D, Raftery AE, Volinsky CT. Bayesian model averaging: a tutorial. Stat Sci. 1999;14(4):382‐401.

Fragoso TM, Bertoli W, Louzada F. Bayesian model averaging: a systematic review and conceptual classification. Int Stat Rev. 2018;86(1):1‐28.

Vevea JL, Hedges LV. A general linear model for estimating effect size in the presence of publication bias. Psychometrika. 1995;60(3):419‐435.

Stanley TD, Doucouliagos H, Ioannidis JP. Finding the power to reduce publication bias. Stat Med. 2017;36(10):1580‐1598.

Erp VS, Verhagen J, Grasman RP, Wagenmakers EJ. Estimates of between‐study heterogeneity for 705 meta‐analyses reported in Psychological Bulletin from 1990–2013. J Open Psychol Data. 2017;5(1):4.

Wrinch D, Jeffreys H. On certain fundamental principles of scientific inquiry. Phil Mag. 1921;42:369‐390.

Kass RE, Raftery AE. Bayes factors. J Am Stat Assoc. 1995;90(430):773‐795.

Jeffreys H. Theory of Probability. 1st ed. Oxford University Press; 1939.

Lee MD, Wagenmakers EJ. Bayesian Cognitive Modeling: A Practical Course. Cambridge University Press; 2013.

Wagenmakers EJ, Morey RD, Lee MD. Bayesian benefits for the pragmatic researcher. Curr Dir Psychol Sci. 2016;25(3):169‐176.

Wasserstein RL, Lazar NA. The ASA statement on p‐values: context, process, and purpose. Am Stat. 2016;70(2):129‐133.

Ioannidis JP, Cappelleri JC, Lau J. Issues in comparisons between meta‐analyses and large trials. Jama. 1998;279(14):1089‐1093.

Laine C, Horton R, DeAngelis CD, et al. Clinical trial registration—looking back and moving ahead. N Engl J Med. 2007;356(26):2734‐2736.

Chambers CD. Registered reports: a new publishing initiative at cortex. Cortex. 2013;49(3):609‐610.

Chambers CD, Dienes Z, McIntosh RD, Rotshtein P, Willmes K. Registered reports: realigning incentives in scientific publishing. Cortex. 2015;66:A1‐A2.

Akker v dOR, Weston S, Campbell L, et al. Preregistration of secondary data analysis: a template and tutorial. Meta‐Psychology. 2021;5:5.

Estimating the false discovery risk of (randomized) clinical trials in medical journals based on published p-values