Meta-analysis assigns more weight to studies with smaller standard errors to maximize the precision of the overall estimate. In observational settings, however, standard errors are shaped by methodological decisions. These decisions can interact with publication bias and p-hacking, potentially leading to spuriously precise results reported by primary studies. Here we show that such spurious precision undermines standard meta-analytic techniques, including inverse-variance weighting and bias corrections based on the funnel plot. Through simulations and large-scale empirical applications, we find that selection models do not resolve the issue. In some cases, a simple unweighted mean of reported estimates outperforms widely used correction methods. We introduce MAIVE (Meta-Analysis Instrumental Variable Estimator), an approach that reduces bias by using sample size as an instrument for reported precision. MAIVE offers a simple and robust solution for improving the reliability of meta-analyses in the presence of spurious precision.

Centre for Economic Policy Research London UK

Institute of Economic Studies Faculty of Social Sciences Charles University Prague Czech Republic

Meta Research Innovation Center at Stanford Stanford University Stanford CA USA

University of Deusto Bilbao Spain

University of the Balearic Islands Palma Spain

See more in PubMed

Gurevitch, J., Koricheva, J., Nakagawa, S. & Stewart, G. Meta-analysis and the science of research synthesis. PubMed

Borenstein, M., Hedges, L., Higgins, J. & Rothstein, H. A basic introduction to fixed-effect and random-effects models for meta-analysis. PubMed

Stanley, T. D. & Doucouliagos, H. Neither fixed nor random: weighted least squares meta-analysis. PubMed

Egger, M., Smith, G. D., Schneider, M. & Minder, C. Bias in meta-analysis detected by a, simple, graphical test. PubMed PMC

Duval, S. & Tweedie, R. Trim and fill: a simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. PubMed

Stanley, T. D. & Doucouliagos, H.

Stanley, T. D. & Doucouliagos, H. Meta-regression approximations to reduce publication selection bias. PubMed

Ioannidis, J. P., Stanley, T. D. & Doucouliagos, H. The power of bias in economics research.

Bom, P. R. D. & Rachinger, H. A kinked meta-regression model for publication bias correction. PubMed

Hedges, L. Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences.

Iyengar, S. & Greenhouse, J. B. Selection models and the file drawer problem.

Hedges, L. V. Modeling publication selection effects in meta-analysis.

Vevea, J. & Hedges, L. V. A general linear model for estimating effect size in the presence of publication bias.

Andrews, I. & Kasy, M. Identification of and correction for publication bias.

van Aert, R. C. & van Assen, M. Correcting for publication bias in a meta-analysis with the p-uniform* method.

Dal-Re, R. et al. Making prospective registration of observational research a reality. PubMed

Bruns, S. B. & Ioannidis, J. P. p-Curve and p-hacking in observational research. PubMed PMC

Abadie, A., Athey, S., Imbens, G. W. & Wooldridge, J. M. When should you adjust standard errors for clustering?

Cameron, A. C. & Miller, D. L. A practitioner’s guide to cluster-robust inference.

Roodman, D., Nielsen, M. Ø., MacKinnon, J. G. & Webb, M. D. Fast and wild: bootstrap inference in stata using boottest.

de Chaisemartin, C. & Ramirez-Cuellar, J. At what level should one cluster standard errors in paired and small-strata experiments?

MacKinnon, J. G., Nielsen, M. O. & Webb, M. D. Cluster-robust inference: a guide to empirical practice.

White, H. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity.

Chesher, A. & Jewitt, I. The bias of a heteroskedasticity consistent covariance matrix estimator.

Lang, K. How credible is the credibility revolution?

Roth, J. Pretest with caution: event-study estimates after testing for parallel trends.

Keane, M. & Neal, T. Instrument strength in IV estimation and inference: a guide to theory and practice.

Young, A. Channeling fisher: randomization tests and the statistical insignificance of seemingly significant experimental results.

Askarov, Z., Doucouliagos, A., Doucouliagos, H. & Stanley, T. D. The significance of data-sharing policy.

Lane, H. & Tranel, B. The Lombard sign and the role of hearing in speech.

Taylor, L. R. Aggregation, variance and the mean.

Stanley, T. D. & Doucouliagos, H. Neither fixed nor random: weighted least squares meta-regression. PubMed

Havranek, T. et al. Reporting guidelines for meta-analysis in economics.

Ugur, M., Awaworyi Churchill, S. & Luong, H. What do we know about R&D spillovers and productivity? Meta-analysis evidence on heterogeneity and statistical power.

Xue, X., Reed, W. R. & Menclova, A. Social capital and health: a meta-analysis. PubMed

Neisser, C. The elasticity of taxable income: a meta-regression analysis.

Nakagawa, S., Lagisz, M. & Jennions, M. D. et al. Methods for testing publication bias in ecological and evolutionary meta-analyses.

Brown, A. L., Imai, T., Vieider, F. & Camerer, C. Meta-analysis of empirical estimates of loss-aversion.

Carter, E. C., Schonbrodt, F. D., Gervais, W. M. & Hilgard, J. Correcting for bias in psychology: a comparison of meta-analytic methods.

Brodeur, A., Cook, N. & Heyes, A. Methods matter: P-hacking and causal inference in economics.

DellaVigna, S. & Linos, E. RCTs to scale: comprehensive evidence from two nudge units.

Imai, T., Rutter, T. A. & Camerer, C. F. Meta-analysis of present-bias estimation using convex time budgets.

Gechert, S. & Heimberger, P. Do corporate tax cuts boost economic growth?

Havranek, T., Irsova, Z., Laslopova, L. & Zeynalova, O. Publication and attenuation biases in measuring skill substitution.

Matousek, J., Havranek, T. & Irsova, Z. Individual discount rates: a meta-analysis of experimental evidence.

Simonsohn, U., Nelson, L. D. & Simmons, J. P. p-Curve: a key to the file-drawer. PubMed

Simonsohn, U., Nelson, L. D. & Simmons, J. P. p-Curve and effect size: correcting for publication bias using only significant results. PubMed

Assen, V. M., Aert, V. R. C. & Wicherts, J. M. Meta-analysis using effect size distributions of only statistically significant studies. PubMed

Simonsohn, U., Simmons, J. P. & Nelson, L. D. Better P-curves: making P-curve analysis more robust to, errors, fraud, and ambitious P-hacking, a Reply to Ulrich and Miller PubMed

Aert, V. R. C. & Assen, V. M. Bayesian evaluation of effect size after replicating an original study. PubMed PMC

Aert, V. R. C. & Assen, V. M. Examining reproducibility in psychology: a hybrid method for combining a statistically significant original study and a replication. PubMed PMC

Stanley, T. D. Beyond publication bias.

Pustejovsky, J. E. & Rodgers, M. A. Testing for funnel plot asymmetry of standardized mean differences. PubMed

Stanley, T. D., Doucouliagos, H. & Havranek, T. Meta-analyses of partial correlations are biased: detection and solutions. PubMed

Kvarven, A., Stromland, E. & Johannesson, M. Comparing meta-analyses and preregistered multiple-laboratory replication projects. PubMed

Bartos, F. et al. Footprint of publication selection bias on meta-analyses in medicine, environmental sciences, psychology, and economics. PubMed

Sanchez-Meca, J. & Martin-Martinez, F. Weighting by inverse variance or by sample size in meta-analysis: a simulation study.

Deeks, J. J., Macaskill, P. & Irwig, L. The performance of tests of publication bias and other sample size effects in systematic reviews. PubMed

Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R. & Rushton, L. Comparison of Two Methods to Detect Publication Bias in Meta-analysis. PubMed

Richard, D. & Riley, L. A. S.

Irsova, Z., Bom, P. R. D., Havranek, T. & Rachinger, H. Replication materials for “Spurious precision in meta-analysis of observational research”. PubMed PMC

Spurious precision in meta-analysis of observational research