Publication bias is a ubiquitous threat to the validity of meta-analysis and the accumulation of scientific evidence. In order to estimate and counteract the impact of publication bias, multiple methods have been developed; however, recent simulation studies have shown the methods' performance to depend on the true data generating process, and no method consistently outperforms the others across a wide range of conditions. Unfortunately, when different methods lead to contradicting conclusions, researchers can choose those methods that lead to a desired outcome. To avoid the condition-dependent, all-or-none choice between competing methods and conflicting results, we extend robust Bayesian meta-analysis and model-average across two prominent approaches of adjusting for publication bias: (1) selection models of p-values and (2) models adjusting for small-study effects. The resulting model ensemble weights the estimates and the evidence for the absence/presence of the effect from the competing approaches with the support they receive from the data. Applications, simulations, and comparisons to preregistered, multi-lab replications demonstrate the benefits of Bayesian model-averaging of complementary publication bias adjustment methods.

Deakin Laboratory for the Meta Analysis of Research Deakin University Melbourne Australia

Department of Economics Deakin University Melbourne Australia

Department of Experimental Psychology University College London London England UK

Department of Psychological Methods University of Amsterdam Amsterdam The Netherlands

Institute of Computer Science Czech Academy of Sciences Prague Czech Republic

Zobrazit více v PubMed

Borenstein M, Hedges L, Higgins J, Rothstein H. Publication Bias. Wiley; 2009:277‐292.

Masicampo E, Lalande DR. A peculiar prevalence of p‐values just below .05. Q J Exp Psychol. 2012;65(11):2271‐2279. PubMed

Scheel AM, Schijen MRMJ, Lakens D. An excess of positive results: comparing the standard psychology literature with registered reports. Adv Methods Pract Psychol Sci. 2021;4(2):1‐12. doi:10.1177/25152459211007467 DOI

Wicherts JM. The weak spots in contemporary science (and how to fix them). Animals. 2017;7(12):90‐119. PubMed PMC

Rothstein HR, Sutton AJ, Borenstein M. Publication Bias in Meta‐Analysis. John Wiley & Sons; 2005.

Rosenthal R, Gaito J. Further evidence for the cliff effect in interpretation of levels of significance. Psychol Rep. 1964;15(2):570.

Simmons JP, Nelson LD, Simonsohn U. False‐positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 2011;22(11):1359‐1366. PubMed

Stefan A, Schönbrodt FD. Big little lies: a compendium and simulation of p‐hacking strategies. PsyArXiv Preprints. 2022. doi:10.31234/osf.io/xy2dk PubMed DOI PMC

John LK, Loewenstein G, Prelec D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci. 2012;23(5):524‐532. PubMed

Fiedler K, Schwarz N. Questionable research practices revisited. Soc Psychol Personal Sci. 2016;7(1):45‐52.

Vevea JL, Hedges LV. A general linear model for estimating effect size in the presence of publication bias. Psychometrika. 1995;60(3):419‐435.

Iyengar S, Greenhouse JB. Selection models and the file drawer problem. Stat Sci. 1988;3(1):109‐117.

Maier M, Bartoš F, Wagenmakers EJ. Robust Bayesian meta‐analysis: addressing publication bias with model‐averaging. Psychol Methods. 2022. PubMed

Egger M, Smith GD, Schneider M, Minder C. Bias in meta‐analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629‐634. PubMed PMC

Stanley TD, Doucouliagos H. Neither fixed nor random: weighted least squares meta‐regression. Res Synth Methods. 2017;8(1):19‐42. PubMed

Stanley TD, Doucouliagos H. Meta‐regression approximations to reduce publication selection bias. Res Synth Methods. 2014;5(1):60‐78. PubMed

Duval S, Tweedie R. Trim and fill: a simple funnel‐plot‐based method of testing and adjusting for publication bias in meta‐analysis. Biometrics. 2000;56(2):455‐463. PubMed

Simonsohn U, Nelson LD, Simmons JP. P‐curve: a key to the file‐drawer. J Exp Psychol Gen. 2014;143(2):534‐547. PubMed

Van Assen MA, Aert VR, Wicherts JM. Meta‐analysis using effect size distributions of only statistically significant studies. Psychol Methods. 2015;20(3):293‐309. PubMed

Andrews I, Kasy M. Identification of and correction for publication bias. Am Econ Rev. 2019;109(8):2766‐2794.

Bom PR, Rachinger H. A kinked meta‐regression model for publication bias correction. Res Synth Methods. 2019;10(4):497‐514. PubMed

Stanley TD, Doucouliagos H, Ioannidis JP. Finding the power to reduce publication bias. Stat Med. 2017;36(10):1580‐1598. PubMed

Copas J. What works?: selectivity models and meta‐analysis. J R Stat Soc A Stat Soc. 1999;162(1):95‐109.

Citkowicz M, Vevea JL. A parsimonious weight function for modeling publication bias. Psychol Methods. 2017;22(1):28‐41. PubMed

Carter EC, Schönbrodt FD, Gervais WM, Hilgard J. Correcting for bias in psychology: a comparison of meta‐analytic methods. Adv Methods Pract Psychol Sci. 2019;2(2):115‐144.

Renkewitz F, Keiner M. How to detect publication bias in psychological research. Z Psychol. 2019;227(4):261‐279.

Hong S, Reed WR. Using Monte Carlo experiments to select meta‐analytic estimators. Res Synth Methods. 2020;12:192‐215. PubMed PMC

Vevea JL, Woods CM. Publication bias in research synthesis: sensitivity analysis using a priori weight functions. Psychol Methods. 2005;10(4):428‐443. PubMed

Rothstein HR, Sutton AJ, Borenstein M. Publication bias in meta‐analysis. Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments; John Wiley & Sons; 2005:1‐7.

Mathur MB, Van der Weele TJ. Sensitivity analysis for publication bias in meta‐analyses. J R Stat Soc Ser C Appl Stat. 2020;69(5):1091‐1119. PubMed PMC

Oswald ME, Grosjean S. Confirmation Bias. Psychology Press; 2004:79‐96.

McShane BB, Böckenholt U, Hansen KT. Adjusting for publication bias in meta‐analysis: an evaluation of selection methods and some cautionary notes. Perspect Psychol Sci. 2016;11(5):730‐749. PubMed

Guan M, Vandekerckhove J. A Bayesian approach to mitigation of publication bias. Psychon Bull Rev. 2016;23(1):74‐86. PubMed

Kvarven A, Strømland E, Johannesson M. Comparing meta‐analyses and preregistered multiple‐laboratory replication projects. Nat Hum Behav. 2020;4(4):423‐434. PubMed

Bem DJ. Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. J Pers Soc Psychol. 2011;100(3):407‐425. PubMed

Hoeting JA, Madigan D, Raftery AE, Volinsky CT. Bayesian model averaging: a tutorial. Stat Sci. 1999;14(4):382‐401.

Leamer EE. Specification Searches: Ad Hoc Inference with Nonexperimental Data. Vol 53. Wiley; 1978.

Hinne M, Gronau QF, van den Bergh D, Wagenmakers EJ. A conceptual introduction to Bayesian model averaging. Adv Methods Pract Psychol Sci. 2020;3(2):200‐215.

Gronau QF, Erp VS, Heck DW, Cesario J, Jonas KJ, Wagenmakers EJ. A Bayesian model‐averaged meta‐analysis of the power pose effect with informed and default priors: the case of felt power. Compr Results Soc Psychol. 2017;2(1):123‐138.

Gronau QF, Heck DW, Berkhout SW, Haaf JM, Wagenmakers EJ. A primer on Bayesian model‐averaged meta‐analysis. Adv Methods Pract Psychol Sci. 2021;4(3):1‐19.

Fragoso TM, Bertoli W, Louzada F. Bayesian model averaging: a systematic review and conceptual classification. Int Stat Rev. 2018;86(1):1‐28.

Jefferys WH, Berger JO. Ockham's razor and Bayesian analysis. Am Sci. 1992;80:64‐72.

Etz A, Wagenmakers EJ. J. B. S. Haldane's contribution to the Bayes factor hypothesis test. Stat Sci. 2017;32:313‐329.

Kass RE, Raftery AE. Bayes factors. J Am Stat Assoc. 1995;90(430):773‐795.

Rouder JN, Morey RD. Teaching Bayes' theorem: strength of evidence as predictive accuracy. Am Stat. 2019;73(2):186‐190.

Wrinch D, Jeffreys H. On certain fundamental principles of scientific inquiry. Philos Mag. 1921;42:369‐390.

Jeffreys H. Theory of Probability. 1st ed. Oxford University Press; 1939.

Lee MD, Wagenmakers EJ. Bayesian Cognitive Modeling: A Practical Course. Cambridge University Press; 2013.

Clyde MA, Ghosh J, Littman ML. Bayesian adaptive sampling for variable selection and model averaging. J Comput Graph Stat. 2011;20(1):80‐101.

Bartoš F, Gronau QF, Timmers B, Otte WM, Ly A, Wagenmakers EJ. Bayesian model‐averaged meta‐analysis in medicine. Stat Med. 2021;40:6743‐6761. PubMed PMC

Wagenmakers EJ, Morey RD, Lee MD. Bayesian benefits for the pragmatic researcher. Curr Dir Psychol Sci. 2016;25:169‐176.

Jeffreys H. Some tests of significance, treated by the theory of probability. Proc Camb Philos Soc. 1935;31:203‐222.

Robinson GK. What properties might statistical inferences reasonably be expected to have? – crisis and resolution in statistical inference. Am Stat. 2019;73:243‐252.

Keysers C, Gazzola V, Wagenmakers EJ. Using Bayes factor hypothesis testing in neuroscience to establish evidence of absence. Nat Neurosci. 2020;23:788‐799. PubMed PMC

Schure TJ, Grünwald P. Accumulation bias in meta‐analysis: the need to consider time in error control. F1000Res. 2019;8:962. PubMed PMC

Hedges LV. Modeling publication selection effects in meta‐analysis. Stat Sci. 1992;7(2):246‐255.

Maier M, Van der Weele TJ, Mathur MB. Using selection models to assess sensitivity to publication bias: a tutorial and call for more routine use. Campbell Syst Rev. 2022;18(3):e1256. PubMed PMC

Copas J, Li H. Inference for non‐random samples. J R Stat Soc Series B Stat Methodology. 1997;59(1):55‐95.

Copas J, Shi JQ. A sensitivity analysis for publication bias in systematic reviews. Stat Methods Med Res. 2001;10(4):251‐265. PubMed

Bem DJ, Utts J, Johnson WO. Must psychologists change the way they analyze their data? J Pers Soc Psychol. 2011;101(4):716‐719. PubMed

Bartoš F, Maier M. RoBMA: An R Package for Robust Bayesian Meta‐Analyses. R package version 2.0.0; 2021. https://CRAN.R-project.org/package=RoBMA

Haaf JM, Rouder JN. Does every study? Implementing ordinal constraint in meta‐analysis. Psychol Methods. 2021. PubMed

Heck, WD , Gronau, FQ , Wagenmakers, EJ . metaBMA: Bayesian Model Averaging for Random and Fixed Effects Meta‐Analysis; 2019. https://CRAN.R-project.org/package=metaBMA

Mathur MB, Van der Weele TJ. Finding common ground in meta‐analysis “wars” on violent video games. Perspect Psychol Sci. 2019;14(4):705‐708. PubMed PMC

Carter EC, McCullough ME. Publication bias and the limited strength model of self‐control: has the evidence for ego depletion been overestimated? Front Psychol. 2014;5:823. PubMed PMC

Moreno SG, Sutton AJ, Turner EH, et al. Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. BMJ. 2009;339:b2981. PubMed PMC

Stanley TD. Limitations of PET‐PEESE and other meta‐analysis methods. Soc Psychol Personal Sci. 2017;8(5):581‐591.

Jeffreys H. Theory of Probability. 3rd ed. Oxford University Press; 1961.

Ritchie SJ, Wiseman R, French CC. Failing the future: three unsuccessful attempts to replicate Bem's ‘Retroactive Facilitation of Recall’ effect. PLoS One. 2012;7(3):e33423. PubMed PMC

Galak J, LeBoeuf RA, Nelson LD, Simmons JP. Correcting the past: failures to replicate psi. J Pers Soc Psychol. 2012;103(6):933‐948. PubMed

Schlitz M, Bem DJ, Marcusson‐Clavertz D, et al. Two replication studies of a time‐reversed (psi) priming task and the role of expectancy in reaction times. J Sci Explor. 2021;35(1):65‐90.

Wagenmakers EJ, Wetzels R, Borsboom D, Maas vHL, Kievit RA. An agenda for purely confirmatory research. Perspect Psychol Sci. 2012;7(6):632‐638. PubMed

Francis G. Too good to be true: publication bias in two prominent studies from experimental psychology. Psychon Bull Rev. 2012;19(2):151‐156. PubMed

Schimmack U. The ironic effect of significant results on the credibility of multiple‐study articles. Psychol Methods. 2012;17(4):551‐566. PubMed

Alcock J. Back from the future: parapsychology and the Bem affair. Skept Inq. 2011;35(2):31‐39.

Hoogeveen S, Sarafoglou A, Wagenmakers EJ. Laypeople can predict which social‐science studies will be replicated successfully. Adv Methods Pract Psychol Sci. 2020;3(3):267‐285.

Wagenmakers EJ, Wetzels R, Borsboom D, Van Der Maas HL. Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011). J Pers Soc Psychol. 2011;100(3):426‐432. PubMed

Schimmack U. Why psychologists should not change the way they analyze their data: The devil is in the default prior. https://replicationindex.com/2015/05/09/why-psychologists-should-not-change-the-way-they-analyze-their-data-the-devil-is-in-the-default-piror/

Schimmack U. My email correspondence with Daryl J. Bem about the data for his 2011 article “Feeling the future”. https://replicationindex.com/2018/01/20/my-email-correspondence-with-daryl-j-bem-about-the-data-for-his-2011-article-feeling-the-future/

Rouder JN, Morey RD. A Bayes factor meta‐analysis of Bem's ESP claim. Psychon Bull Rev. 2011;18(4):682‐689. PubMed

Aczel B, Palfi B, Szollosi A, et al. Quantifying support for the null hypothesis in psychology: an empirical investigation. Adv Methods Pract Psychol Sci. 2018;1(3):357‐366.

Goodman S. A dirty dozen: twelve p‐value misconceptions. Semin Hematol. 2008;45:135‐140. PubMed

Maier M, Bartoš F, Oh M, Wagenmakers EJ, Shanks D, Harris A. Publication bias in research on construal level theory. PsyArXiv Preprints. 2022. doi:10.31234/osf.io/r8nyu DOI

Maier M, Bartoš F, Stanley TD, Shanks DR, Harris JLA, Wagenmakers EJ. No evidence for nudging after adjusting for publication bias. Proc Natl Acad Sci. 2022;119:e2200300119. PubMed PMC

Graham J, Haidt J, Nosek BA. Liberals and conservatives rely on different sets of moral foundations. J Pers Soc Psychol. 2009;96(5):1029‐1046. PubMed

Kivikangas JM, Fernández‐Castilla B, Järvelä S, Ravaja N, Lönnqvist JE. Moral foundations and political orientation: systematic review and meta‐analysis. Psychol Bull. 2021;147(1):55‐94. PubMed

Klein RA, Vianello M, Hasselman F, et al. Many labs 2: investigating variation in replicability across samples and settings. Adv Methods Pract Psychol Sci. 2018;1(4):443‐490.

Alinaghi N, Reed WR. Meta‐analysis and publication bias: how well does the FAT‐PET‐PEESE procedure work? Res Synth Methods. 2018;9(2):285‐311. PubMed

Ioannidis JP, Baas J, Klavans R, Boyack KW. A standardized citation metrics author database annotated for scientific field. PLoS Biol. 2019;17(8):e3000384. PubMed PMC

Bartoš F, Maier M, Quintana D, Wagenmakers EJ. Adjusting for publication bias in JASP and R‐selection models, PET‐PEESE, and robust Bayesian meta‐analysis. Adv Methods Pract Psychol Sci. in press.

Gronau QF, Ly A, Wagenmakers EJ. Informed Bayesian t‐tests. Am Stat. 2020;74(2):137‐143.

JASP Team . JASP (Version 0.15) [Computer software]; 2021. https://jasp-stats.org/

Ly A, van den Bergh D, Bartoš F, Wagenmakers EJ. Bayesian inference with JASP. ISBA Bull. 2021;28:7‐15.

Estimating the false discovery risk of (randomized) clinical trials in medical journals based on published p-values

Informed Bayesian survival analysis