The number of biomedical image analysis challenges organized per year is steadily increasing. These international competitions have the purpose of benchmarking algorithms on common data sets, typically to identify the best method for a given problem. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results. To address the discrepancy between the impact of challenges and the quality (control), the Biomedical Image Analysis ChallengeS (BIAS) initiative developed a set of recommendations for the reporting of challenges. The BIAS statement aims to improve the transparency of the reporting of a biomedical image analysis challenge regardless of field of application, image modality or task category assessed. This article describes how the BIAS statement was developed and presents a checklist which authors of biomedical image analysis challenges are encouraged to include in their submission when giving a paper on a challenge into review. The purpose of the checklist is to standardize and facilitate the review process and raise interpretability and reproducibility of challenge results by making relevant information explicit.

Centre for Biomedical Image Analysis Masaryk University Botanická 68a Brno 60200 Czech Republic

Centre for Intelligent Machines McGill University 3480 University Street McConnell Engineering Building Room 425 Montreal QC H3A 0E9 Canada

Department of Radiology and Nuclear Medicine Medical Image Analysis Radboud University Center Nijmegen 6525 GA The Netherlands

Division of Biostatistics German Cancer Research Center Im Neuenheimer Feld 581 Heidelberg 69120 Germany

Division of Computer Assisted Medical Interventions Im Neuenheimer Feld 223 Heidelberg 69120 Germany

Electrical Engineering Vanderbilt University Nashville Tennessee TN 37235 1679 USA

Institute of Computational Biomedicine Heidelberg University Faculty of Medicine Im Neuenheimer Feld 267 Heidelberg 69120 Germany; Heidelberg University Hospital Im Neuenheimer Feld 267 Heidelberg 69120 Germany; Joint Research Centre for Computational Biomedicine Rheinisch Westfälische Technische Hochschule Aachen Faculty of Medicine Aachen 52074 Germany

Institute of Information Systems Engineering Technische Universität Wien Favoritenstraße 9 11 194 04 Vienna 1040 Austria; Complexity Science Hub Vienna Josefstädter Straße 39 Vienna 1080 Austria

Laboratoire Traitement du Signal et de l'Image UMR_S 1099 Université de Rennes 1 Inserm Rennes Cedex 35043 France

Physical Sciences Sunnybrook Research Institute 2075 Bayview Avenue Rm M6 609 Toronto ON M4N 3M5 Canada; Department Medical Biophysics University of Toronto 101 College St Suite 15 701 Toronto ON M5G 1L7 Canada

University of Applied Sciences Western Switzerland Rue du Technopole 3 Sierre 3960 Switzerland; Medical Faculty University of Geneva Rue Gabrielle Perret Gentil 4 Geneva 1211 Switzerland

Zobrazit více v PubMed

Ayache N., Duncan J. 20th anniversary of the medical image analysis journal (MedIA) Med. Image Anal. 2016;33:1–3. doi: 10.1016/j.media.2016.07.004. PubMed DOI

Bora K., Chowdhury M., Mahanta L.B., Kundu M.K., Das A.K. Automated classification of pap smear images to detect cervical dysplasia. Comput. Methods Programs Biomed. 2017;138:31–47. doi: 10.1016/j.cmpb.2016.10.001. PubMed DOI

Bychkov D., Linder N., Turkki R., Nordling S., Kovanen P.E., Verrill C., Walliander M., Lundin M., Haglund C., Lundin J. Deep learning based tissue analysis predicts outcome in colorectal cancer. Sci. Rep. 2018;8(1):3395. doi: 10.1038/s41598-018-21758-3. PubMed DOI PMC

Chenouard N., Smal I., De Chaumont F., Maška M., Sbalzarini I.F., Gong Y., Cardinale J., Carthel C., Coraluppi S., Winter M., Cohen A.R., Godinez W.J., Rohr K., Kalaidzidis Y., Liang L., Duncan J., Shen H., Xu Y., Magnusson K.E.G., Jaldé, Blau H.M., Paul-Gilloteaux P., Roudot P., Kervrann C., Waharte F., Tinevez J.-Y., Shorte S.L., Willemse J., Celler K., van Wezel G.P., Dan H.-W., Tsai Y.-S., Ortiz de Solórzano C., Olivo-Marin J.-C., Meijering E. Objective comparison of particle tracking methods. Nat. Methods. 2014;11(3):281–289. doi: 10.1038/nmeth.2808. PubMed DOI PMC

crowdAI, 2018. crowdAI. www.crowdai.org/. Accessed: 2019-04-01.

Daneshjou R., Wang Y., Bromberg Y., Bovo S., Martelli P.L., Babbi G., Lena P.D., Casadio R., Edwards M., Gifford D., Jones D.T., Sundaram L., Bhat R.R., Li X., Kundu K., Yin Y., Moult J., Jiang Y., Pejaver V., Pagel K.A., Li B., Mooney S.D., Radivojac P., Shah S., Carraro M., Gasparini A., Leonardi E., Giollo M., Ferrari C., Tosatto S.C.E., Bachar E., Azaria J.R., Ofran Y., Unger R., Niroula A., Vihinen M., Chang B., Wang M.H., Franke A., Petersen B.S., Prooznia M., Zandi P., McCombie R., Potash J.B., Altman R.B., Klein T.E., Hoskins R.A., Repo S., Brenner S.E., Morgan A.A. Working toward precision medicine: Predicting phenotypes from exomes in the critical assessment of genome interpretation (CAGI) challenges. Hum. Mut. 2017;38(9):1182–1192. doi: 10.1002/humu.23280. PubMed DOI PMC

Doğan R.I., Kim S., Chatr-aryamontri A., Wei C.-H., Comeau D.C., Antunes R., Matos S., Chen Q., Elangovan A., Panyam N.C., Verspoor K., Liu H., Wang Y., Liu Z., Altinel B., Hsnbeyi Z.M., Zgr A., Fergadis A., Wang C.-K., Dai H.-J., Tran T., Kavuluru R., Luo L., Steppi A., Zhang J., Qu J., Lu Z. Overview of the BioCreative VI precision medicine track: mining protein interactions and mutations for precision medicine. Database (Oxford) 2019;2019 doi: 10.1038/10.1093/database/bay147. PubMed DOI PMC

DREAM, 2006. Dream challenges. dreamchallenges.org/. Accessed: 2019-04-01.

Esteva A., Robicquet A., Ramsundar B., Kuleshov V., DePristo M., Chou K., Cui C., Corrado G., Thrun S., Dean J. A guide to deep learning in healthcare. Nat. Med. 2019;25(1):24. doi: 10.1038/s41591-018-0316-z. PubMed DOI

Guinney J., Saez-Rodriguez J. Alternative models for sharing confidential biomedical data. Nat. Biotechnol. 2018;36(5):391. doi: 10.1038/nbt.4128. PubMed DOI

Hosny A., Parmar C., Quackenbush J., Schwartz L.H., Aerts H.J. Artificial intelligence in radiology. Nat. Rev. Cancer. 2018;18:500. doi: 10.1038/s41568-018-0016-5. PubMed DOI PMC

Jannin P., Grova C., Maurer C.R. Model for defining and reporting reference-based validation protocols in medical image processing. Int. J. CARS. 2006;1(2):63–73. doi: 10.1007/s11548-006-0044-6. DOI

Kaggle, 2010. Your home for data science. www.kaggle.com/. Accessed: 2019-02-12.

Kohl S., Romera-Paredes B., Meyer C., De Fauw J., Ledsam J.R., Maier-Hein K., Eslami S.A., Rezende D.J., Ronneberger O. Adv. Neural Inf. Process. Syst. (NIPS 2018) Vol. 31. 2018. A probabilistic U-net for segmentation of ambiguous images; pp. 6965–6975.arXiv:1806.05034

Kozubek M. Focus Bio Image Inform. Springer; 2016. Challenges and benchmarks in bioimage analysis; pp. 231–262. PubMed DOI

Litjens G., Kooi T., Bejnordi B.E., Setio A.A.A., Ciompi F., Ghafoorian M., Van Der Laak J.A., Van Ginneken B., Sánchez C.I. A survey on deep learning in medical image analysis. Med. Image Anal. 2017;42:60–88. doi: 10.1016/j.media.2017.07.005. PubMed DOI

Maier-Hein K.H., Neher P.F., Houde J.-C., Côté M.-A., Garyfallidis E., Zhong J., Chamberland M., Yeh F.-C., Lin Y.-C., Ji Q., Reddick W.E., Glass J.O., Qixiang Chen D., Feng Y., Gao C., Wu Y., Ma J., Renjie H., Li Q., Westin C.-F., Deslauriers-Gauthier S., Omar Ocegueda González J., Paquette M., St-Jean S., Girard G., Rheault F., Sidhu J., Tax C.M.W., Guo F., Mesri H.Y., Dávid S., Froeling M., Heemskerk A.M., Leemans A., Boré A., Pinsard B., Bedetti C., Desrosiers M., Brambati S., Doyon J., Sarica A., Vasta R., Cerasa A., Quattrone A., Yeatman J., Khan A.R., Hodges W., Alexander S., Romascano D., Barakovic M., Auría A., Esteban O., Lemkaddem A., Thiran J.-P., Cetingul H.E., Odry B.L., Mailhe B., Nadar M.S., Pizzagalli F., Prasad G., Villalon-Reina J.E., Galvis J., Thompson P.M., De Santiago Requejo F., Laguna P.L., Lacerda L.M., Barrett R., Dell’cqua F., Catani M., Petit L., Caruyer E., Daducci A., Dyrby T.B., Holland-Letz T., Hilgetag C.C., Bram S., Descoteaux M. The challenge of mapping the human connectome based on diffusion tractography. Nat. Commun. 2017;8(1):1349. doi: 10.1038/s41467-017-01285-x. PubMed DOI PMC

Maier-Hein L., Eisenmann M., Reinke A., Onogur S., Stankovic M., Scholz P., Arbel T., Bogunovic H., Bradley A.P., Carass A., Feldmann C., Frangi A.F., Full P.M., van Ginneken B., Hanbury A., Honauer K., Kozubek M., Landman B.A., Mrz K., Maier O., Maier-Hein K., Menze B.H., Müller H., Neher P.F., Niessen W., Rajpoot N., Sharp G.C., Sirinukunwattana K., Speidel S., Stock C., Stoyanov D., Taha A.A., van der Sommen F., Wang C.-W., Weber M.-A., Zheng G., Jannin P., Kopp-Schneider A. Why rankings of biomedical image analysis competitions should be interpreted with care. Nat. Commun. 2018;9(1):5217. doi: 10.1038/s41467-019-08563-w. PubMed DOI PMC

Maier-Hein L., Mersmann S., Kondermann D., Stock C., Kenngott H.G., Sanchez A., Wagner M., Preukschas A., Wekerle A.-L., Helfert S., Bodenstedt S., Speidel S. Int. Conf. Med. Image Comput. Comp. Assis. Interv. Springer; 2014. Crowdsourcing for reference correspondence generation in endoscopic images; pp. 349–356. PubMed DOI

Maier-Hein L., Reinke A., Kozubek M., Martel A.L., Arbel T., Eisenmann M., Hanbury A., Jannin P., Müller H., Onogur S., Saez-Rodriguez J., van Ginneken B., Kopp-Schneider A., Landman B.A. Biomedical Image Analysis Challenges (BIAS) Reporting Guideline. Zenodo. 2020 doi: 10.5281/zenodo.4008953. PubMed DOI PMC

Meldrum M.L. A brief history of the randomized controlled trial: From oranges and lemons to the gold standard. Hematol. Oncol. Clin. North Am. 2000;14(4):745–760. doi: 10.1016/S0889-8588(05)70309-9. PubMed DOI

Menze B.H., Jakab A., Bauer S., Kalpathy-Cramer J., Farahani K., Kirby J., Burren Y., Porz N., Slotboom J., Wiest R., Lanczi L., Gerstner E., Weber M.-A., Arbel T., Avants B.B., Ayache N., Buendia P., Collins D.L., Cordier N., Corso J.J., Criminisi A., Das T., Delingette H., Demiralp C., Durst C.R., Dojat M., Doyle S., Festa J., Forbes F., Geremia E. The multimodal brain tumor image segmentation benchmark (BRATS) IEEE Trans. Med. Imaging. 2015;34(10):1993–2024. doi: 10.1109/TMI.2014.2377694. PubMed DOI PMC

Moher D., Schulz K.F., Altman D.G. The consort statement: revised recommendations for improving the quality of reports of parallel group randomized trials. BMC Med. Res. Methodol. 2001;1(1):2. PubMed PMC

Moult J., Fidelis K., Kryshtafovych A., Schwede T., Tramontano A. Critical assessment of methods of protein structure prediction (CASP) – Round XII. Proteins. 2017;86:7–15. doi: 10.1002/prot.25415. PubMed DOI PMC

Saez-Rodriguez J., Costello J.C., Friend S.H., Kellen M.R., Mangravite L., Meyer P., Norman T., Stolovitzky G. Crowdsourcing biomedical research: leveraging communities as innovation engines. Nat. Rev. Genet. 2016;17(8):470–486. doi: 10.1038/nrg.2016.69. PubMed DOI PMC

Sage D., Kirshner H., Pengo T., Stuurman N., Min J., Manley S., Unser M. Quantitative evaluation of software packages for single-molecule localization microscopy. Nat. Methods. 2015;12(8):717–724. doi: 10.1038/nmeth.3442. PubMed DOI

Schulz K.F., Altman D.G., Moher D. Consort 2010 statement: updated guidelines for reporting parallel group randomised trials. BMC Med. 2010;8(1):18. PubMed PMC

Setio A.A.A., Traverso A., De Bel T., Berens M.S., van den Bogaard C., Cerello P., Chen H., Dou Q., Fantacci M.E., Geurts B., van der Gugten R., Heng P.A., Jansen B., de Kaste M.M.J., Kotov V., Lin J.Y.-H., Manders J.T.M.C., Sóñora Mengana A., García-Naranjo J.C., Papavasileiou E., Prokop M., Saletta M., Schaefer-Prokop C.M., Scholten E.T., Scholten L., Snoeren M.M., Torres E.L., Vandemeulebroucke J., Walasek N., Zuidhof G.C.A., van Ginneken B., Jacobs C. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge. Med. Image Anal. 2017;42:1–13. doi: 10.1016/j.media.2017.06.015. PubMed DOI

TheEQUATORNetwork, 2008. The EQUATOR network – Enhancing the QUAlity and Transparency Of health Research. http://www.equator-network.org. Accessed: 2019-09-12.

Topol E.J. High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 2019;25(1):44. doi: 10.1038/s41591-018-0300-7. PubMed DOI

Wiesenfarth M., Reinke A., Landman B.A., Cardoso M.J., Maier-Hein L., Kopp-Schneider A. Methods and open-source toolkit for analyzing and visualizing challenge results. arXiv preprint arXiv:1910.05121. 2019 Submitted for publication. PubMed PMC

Yu K.-H., Zhang C., Berry G.J., Altman R.B., Ré C., Rubin D.L., Snyder M. Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nat. Commun. 2016;7:12474. doi: 10.1038/ncomms12474. PubMed DOI PMC

Zenodo, 2013. https://zenodo.org/(accessed: June 2020).

Zheng G., Chu C., Belavỳ D.L., Ibragimov B., Korez R., Vrtovec T., Hutt H., Everson R., Meakin J., Andrade I.L.A., Glocker B., Chen H., Dou Q., Heng P.-A., Qang C., Forsberg D., Neubert A., Fripp J., Urschler M., Stern D., Wimmer M., Novikov A.A., Cheng H., Armbrecht G., Felsenberg D., Li S. Evaluation and comparison of 3D intervertebral disc localization and segmentation methods for 3D T2 MR data: A grand challenge. Med. Image Anal. 2017;35:327–344. doi: 10.1016/j.media.2016.08.005. PubMed DOI

BIAS: Transparent reporting of biomedical image analysis challenges