Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well-justified, systematic and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of individuals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target. The resulting checklist can be used for transparently communicating the rationale for selecting studies for replication.

Centre for Contextual Behavioural Science School of Psychology University of Chester Chester UK

Centre of Science and Technology Studies Leiden University Leiden the Netherlands

Department of Cognition Emotion and Methods in Psychology Faculty of Psychology University of Vienna Vienna Austria

Department of Methodology and Statistics Tilburg University Tilburg The Netherlands

Department of Preventive Medicine Division of Biostatistics Feinberg School of Medicine Northwestern University Chicago IL USA

Department of Psychology Education and Child Studies Erasmus University Rotterdam Rotterdam The Netherlands

Department of Psychology University of Gothenburg Gothenburg Sweden

Department of Psychology University of Toronto Toronto Canada

Department of Psychometrics and Statistics Rijksuniversiteit Groningen Groningen The Netherlands

Faculty of Social Sciences Charles University Prague Czech Republic

Institute for Interdisciplinary Brain and Behavioral Sciences Chapman University Orange CA USA

Institute of Applied Health Research University of Birmingham Birmingham UK

Maison de santé pluridisciplinaire Pasteur Chevilly Larue France

Methodology and Statistics Unit Institute of Psychology Leiden University Leiden the Netherlands

Oslo New University College Oslo Norway

Philosophy Department and the Faculty of Religion and Theology Vrije Universiteit Amsterdam The Netherlands

Program in Cognitive Science Department of Instructional Systems Technology Indiana University Bloomington IN USA

Research Fellow of the Japan Society for the Promotion of Science Erasmus University Rotterdam Rotterdam The Netherlands

Research Organization of Open Innovation and Collaboration Ritsumeikan University Osaka Japan

School of Education Language and Psychology York St John University York UK

School of Psychology University of Kent Canterbury UK

Social Psychology University of Cologne Cologne Germany

WellStar College of Health and Human Services Kennesaw State University Kennesaw GA USA

Zobrazit více v PubMed

Pashler H, Wagenmakers E. 2012. Editor’s introduction to the special section on replicability in psychological science: a crisis of confidence?. Perspect. Psychol. Sci. 7, 528-530. ( 10.1177/1745691612465253) PubMed DOI

Baker M. 2016. Dutch agency launches first grants programme dedicated to replication. Nature 2016. ( 10.1038/nature.2016.20287) DOI

Ioannidis JPA. 2005. Why most published research findings are false. PLoS Med. 2, e124. ( 10.1371/journal.pmed.0020124) PubMed DOI PMC

Levelt Committee, Noort Committee, Drenth Committee. 2012 Flawed science: the fraudulent research practices of social psychologist Diederik Stapel. Technical report.

Wagenmakers EJ, Wetzels R, Borsboom D, van der Maas HL, Kievit RA. 2012. An agenda for purely confirmatory research. Perspect. Psychol. Sci. 7, 632-638. ( 10.1177/1745691612463078) PubMed DOI

Simmons JP, Nelson LD, Simonsohn U. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359-1366. ( 10.1177/0956797611417632) PubMed DOI

Fiedler K. 2011. Voodoo correlations are everywhere-not only in neuroscience. Perspect. Psychol. Sci.: A J. Assoc. Psychol. Sci. 6, 163-71. ( 10.1177/1745691611400237) PubMed DOI

Makel MC, Plucker JA, Hegarty B. 2012. Replications in psychology research: How often do they really occur? Perspect. Psychol. Sci. 7, 537-542. ( 10.1177/1745691612460688) PubMed DOI

Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349, aac4716. ( 10.1126/science.aac4716) PubMed DOI

Isager PM, van't Veer A, Nosten T, Janson E, Lakens D. 2019. Quantifying Replication Value: A guide in the decision of what to replicate.

McNeeley S, Warner JJ. 2015. Replication in criminology: a necessary practice. Eur. J. Criminol. 12, 581-597. ( 10.1177/1477370815578197) DOI

Errington TM, Iorns E, Gunn W, Tan FE, Lomax J, Nosek BA. 2014. An open investigation of the reproducibility of cancer biology research. eLife 3, e04333. ( 10.7554/eLife.04333) PubMed DOI PMC

Cook SC, Schwartz AC, Kaslow NJ. 2017. Evidence-based psychotherapy: advantages and challenges. Neurotherapeutics 14, 537-545. ( 10.1007/s13311-017-0549-4) PubMed DOI PMC

Institure of Educational Sciences. 2019 IES FY 2020 Request for Applications Research Grants Focused on Systematic Replication CFDA Number: 84.305R. Technical report.

Berlin Institute of Health, QUEST Center. 2018. The QUEST Replication Call. https://www.bihealth.org/fileadmin/QUEST/Publikationen/QUEST_replication_funding_scheme.pdf.

Isager PM, Schwartz AC, Kaslow NJ. 2021. Deciding what to replicate: a decision model for replication study selection under resource and knowledge constraints, Psychol. Methods. PubMed

Laws KR. 2016. Psychology, replication & beyond. BMC Psychol. 4, 30-30. ( 10.1186/s40359-016-0135-2) PubMed DOI PMC

Field SM, Hoekstra R, Bringmann L, van Ravenzwaaij D. 2019. When and why to replicate: as easy as 1, 2, 3? Collabra: Psychol. 5, 46-46. ( 10.1525/collabra.218) DOI

Pittelkow MM, Hoekstra R, Karsten J, van Ravenzwaaij D. 2021. Replication target selection in clinical psychology: a Bayesian and qualitative reevaluation. Clin. Psychol.: Sci. Practice 28, 210-221. ( 10.1037/cps0000013) DOI

Coles NA, Tiokhin L, Scheel AM, Isager PM, Lakens D. 2018. The costs and benefits of replication studies. Behav. Brain Sci. 41, e124. ( 10.1017/S0140525X18000596) PubMed DOI

Hardwicke TE, Tessler MH, Peloquin BN, Frank MC. 2018. A Bayesian decision-making framework for replication. Behav. Brain Sci. 41, e132-e132. ( 10.1017/S0140525X18000675) PubMed DOI

Kuehberger A, Schulte-Mecklenbeck M. 2018. Selecting target papers for replication. Behav. Brain Sci. 41, e139. ( 10.1017/S0140525X18000742) PubMed DOI

Nuijten M. 2021. Efficient scientific self-correction in times of crisis. In The new common: How the COVID-19 pandemic is transforming society (eds Aaarts E, Fleuren H, Sitskoorn M, Wilthagen T), pp. 161-167. London: Springer Nature.

Murphy J, Mesquida C, Caldwell AR, Earp BD, Warne JP. 2023. Proposal of a selection protocol for replication of studies in sports and exercise science. Sports medicine 53, 281-297. ( 10.1007/s40279-022-01749-1) PubMed DOI PMC

Waggoner J, Carline JD, Durning SJ. 2016. Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research. Acad. Med. 91, 663-668. ( 10.1097/ACM.0000000000001092) PubMed DOI

McKenna HP. 1994. The Delphi technique: a worthwhile research approach for nursing? J. Adv. Nurs. 19, 1221-1225. ( 10.1111/j.1365-2648.1994.tb01207.x) PubMed DOI

LeBel EP, McCarthy RJ, Earp BD, Elson M, Vanpaemel W. 2018. A unified framework to quantify the credibility of scientific findings. Adv. Methods Pract. Psychol. Sci. 1, 389-402. ( 10.1177/2515245918787489) DOI

Höffmeier J, Mazei J, Schultze T. 2016. Reconceptualizing replication as a sequence of different studies: a replication typology. J. Exp. Soc. Psychol. 66, 81-92. ( 10.1016/j.jesp.2015.09.009) DOI

Muradchanian J, Hoekstra R, Kiers H, van Ravenzwaaij D. 2021. How best to quantify replication success? A simulation study on the comparison of replication success metrics. R. Soc. Open Sci. 8, 201697. ( 10.1098/rsos.201697) PubMed DOI PMC

Field SM, Wagenmakers EJ, Newell BR, Zeelenberg R, van Ravenzwaaij D. 2016. Two Bayesian tests of the GLOMOsys Model. J. Exp. Psychol. Gen. 145, e81-e95. ( 10.1037/xge0000067) PubMed DOI

van Ravenzwaaij D, Boekel W, Forstmann BU, Ratcliff R, Wagenmakers EJ. 2014. Action video games do not improve the speed of information processing in simple perceptual tasks. J. Exp. Psychol. Gen. 143, 1794. ( 10.1037/a0036923) PubMed DOI PMC

Brandt MJ, et al. 2014. The replication recipe: what makes for a convincing replication?. J. Exp. Soc. Psychol. 50, 217-224. ( 10.1016/j.jesp.2013.10.005) DOI

Klein RA, et al. 2018. Many labs 2: investigating variation in replicability across samples and settings Adv. Methods Pract. Psychol. Sci. 1, 443-490. ( 10.1177/2515245918810225) DOI

Collaboration OS. 2012. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspect. Psychol. Sci. 7, 657-660. (doi:1177/1745691612462588) PubMed

Bouwmeester S, et al. 2017. Registered replication report: rand, greene, and Nowak. Perspect. Psychol. Sci. 12, 527-542. ( 10.1177/1745691617693624) PubMed DOI PMC

Field SM, Wagenmakers EJ, Kiers HAL, Hoekstra R, Ernst AF, van Ravenzwaaij D. 2020. The effect of preregistration on trust in empirical research findings: results of a registered report. R. Soc. Open Sci. 7, 181351. ( 10.1098/rsos.181351) PubMed DOI PMC

Alister M, Vickers-Jones R, Sewell DK, Ballard T. 2021. How do we choose our giants? Perceptions of replicability in psychological science. Adv. Methods Pract. Psychol. Sci. 4, 25152459211018199. (doi:10.1177/25152459211018199) DOI

Aczel B, et al. 2020. A consensus-based transparency checklist. Nat. Hum. Behav. 4, 4-6. ( 10.1038/s41562-019-0772-6) PubMed DOI PMC

Braun V, Clarke V. 2006. Using thematic analysis in psychology. Qual. Res. Psychol. 3, 77-101. ( 10.1191/1478088706qp063oa) DOI

Field SM, Ravenzwaaij D, Pittelkow M-M, Hoek J M, Derksen M. 2021. Qualitative open science - pain points and perspective. ( 10.31219/osf.io/e3cq4) DOI

Miles MB, Huberman AM. 1994. Qualitative data analysis: an expanded sourcebook. Thousand Oaks: CA: Sage.

Syed M, Nelson SC. 2015. Guidelines for establishing reliability when coding narrative data. Emerg. Adulthood 3, 375-387. ( 10.1177/2167696815587648) DOI

Birko S, Dove ES, Özdemir V. 2015. Evaluation of nine consensus indices in Delphi foresight research and their dependency on Delphi survey characteristics: a simulation study and debate on Delphi design and interpretation. PLoS ONE 10, e0135162. ( 10.1371/journal.pone.0135162) PubMed DOI PMC

Noy C. 2008. Sampling knowledge: the hermeneutics of snowball sampling in qualitative research. Int. J. Soc. Res. Methodol. 11, 327-344. ( 10.1080/13645570701401305) DOI

Venette S. 2013. What is snowball sampling?. (https://www.researchgate.net/post/What_is_snowball_sampling#:~:text=Popular%20answers%20(1)-,4th%20Apr%2C%202013,recruits%20them%20for%20the%20study.)

Jorm AF. 2015. Using the Delphi expert consensus method in mental health research. Aust. N. Zeal. J. Psychiatry 49, 887-897. ( 10.1177/0004867415600891) PubMed DOI

Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J, Marteau T. 1998. Consensus development methods, and their use in clinical guideline development. Health Technol. Assess. (Winchester, England) 2, i-88. ( 10.3310/hta2030) PubMed DOI

Gargon E, Crew R, Burnside G, Williamson PR. 2019. Higher number of items associated with significantly lower response rates in COS Delphi surveys. J. Clin. Epidemiol. 108, 110-120. ( 10.1016/j.jclinepi.2018.12.010) PubMed DOI PMC

Braun V, Clarke V. 2022. Thematic analysis: a practical guide. Los Angeles, CA, USA: SAGE.

Clemens MA. 2017. The meaning of failed replications: a review and proposal. J. Econ. Surv. 31, 326-342. ( 10.1111/joes.12139) DOI

Zwaan RA, Etz A, Lucas RE, Donnellan MB. 2018. Improving social and behavioral science by making replication mainstream: a response to commentaries. Behav. Brain Sci. 41, e157-e157. ( 10.1017/S0140525X18000961) PubMed DOI

Gómez OS, Juristo N, Vegas S. 2014. Understanding replication of experiments in software engineering: a classification, Inf. Softw. Technol. 56, 1033-1048. ( 10.1016/j.infsof.2014.04.004) DOI

Isager PM. 2018. What to Replicate? Justifications of study choice from 85 replication studies. Technical report. Zenodo.

Derksen M, Field S. 2022. The tone debate: knowledge, self, and social order. Rev. Gen. Psychol. 26, 172-183. ( 10.1177/10892680211015636) DOI

Field SM, Derksen M. 2021. Experimenter as automaton; experimenter as human: exploring the position of the researcher in scientific research. Eur. J. Philos. Sci. 11, 1-21. ( 10.1007/s13194-020-00324-7) DOI

Bauer G, Breznau N, Gereke J, Höffler JH, Janz N, Rahal RM, Rennstich JK, Soiné H. 2022. Teaching constructive replications in the social sciences. ( 10.31222/osf.io/ejkws) DOI

Schmidt S. 2009. Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Rev. Gen. Psychol. 13, 90-100. ( 10.1037/a0015108) DOI

Zwaan RA, Etz A, Lucas RE, Donnellan MB. 2018. Making replication mainstream. Behav. Brain Sci. 41, e120. ( 10.1017/S0140525X17001972) PubMed DOI

Soderberg CK, Errington TM. 2019. Replications and the social and behavioral sciences. In

LeBel EP, Berger D, Campbell L, Loving TJ. 2017. Falsifiability is not optional. J. Pers. Soc. Psychol. 113, 254-261. ( 10.1037/pspi0000106) PubMed DOI

Wuestefeld A, et al. 2020. Towards reporting guidelines of research using whole-body vibration as training or treatment regimen in human subjects–A Delphi consensus study. PLoS ONE 15, e0235905. ( 10.1371/journal.pone.0235905) PubMed DOI PMC

Pittelkow M-M, et al. 2023. The process of replication target selection in psychology: what to consider? Figshare. ( 10.6084/m9.figshare.c.6403449) PubMed DOI PMC

The process of replication target selection in psychology: what to consider?

Zobrazit více v PubMed

figshare
10.6084/m9.figshare.c.6403449