Image analysis is key to extracting quantitative information from scientific microscopy images, but the methods involved are now often so refined that they can no longer be unambiguously described by written protocols. We introduce BIAFLOWS, an open-source web tool enabling to reproducibly deploy and benchmark bioimage analysis workflows coming from any software ecosystem. A curated instance of BIAFLOWS populated with 34 image analysis workflows and 15 microscopy image datasets recapitulating common bioimage analysis problems is available online. The workflows can be launched and assessed remotely by comparing their performance visually and according to standard benchmark metrics. We illustrated these features by comparing seven nuclei segmentation workflows, including deep-learning methods. BIAFLOWS enables to benchmark and share bioimage analysis workflows, hence safeguarding research results and promoting high-quality standards in image analysis. The platform is thoroughly documented and ready to gather annotated microscopy datasets and workflows contributed by the bioimaging community.

Cytomine SCRL FS 4000 Liège Belgium

Dundee Imaging Facility School of Life Sciences University of Dundee Dundee DD1 5EH UK

Faculty of Engineering İzmir Demokrasi University 35330 Balçova Turkey

FIMM HiLIFE University of Helsinki 00014 Helsinki Finland

HEPL University of Liège 4000 Liège Belgium

Institute for Research in Biomedicine IRB Barcelona Barcelona Institute of Science and Technology BIST 08028 Barcelona Spain

Life Sciences Core Facilities Weizmann Institute of Science Rehovot 7610001 Israel

Masaryk University 601 77 Brno Czech Republic

Montefiore Institute University of Liège 4000 Liège Belgium

MRI BioCampus Montpellier Montpellier 34094 France

Politehnica Bucarest Bucarest 060042 Romania

Structure Fédérative de Recherche François Bonamy Université de Nantes CNRS INSERM Nantes Cedex 1 13522 44035 France

Universidade Federal do Paraná Curitiba 80060 000 Brazil

Uppsala University P O Box 256 751 05 Uppsala Sweden

VIB BioImaging Core 3000 Leuven Belgium

Zobrazit více v PubMed

Ouyang W., Zimmer C. The imaging tsunami: computational opportunities and challenges. Curr. Opin. Syst. Biol. 2017;4:105–113.

Eliceiri K.W., Berthold M.R., Goldberg I.G., Ibáñez L., Manjunath B.S., Martone M.E., Murphy R.F., Peng H., Plant A.L., Roysam B. Biological imaging software tools. Nat. Methods. 2012;9:697–710. PubMed PMC

Carpenter A.E., Kamentsky L., Eliceiri K.W. A call for bioimaging software usability. Nat. Methods. 2012;9:666–670. PubMed PMC

Schneider C.A., Rasband W.S., Eliceiri K.W. NIH Image to ImageJ: 25 years of image analysis. Nat. Methods. 2012;9:671–675. PubMed PMC

Munafò M.R., Nosek B.A., Bishop D.V.M., Button K.S., Chambers C.D., Percie du Sert N., Simonsohn U., Wagenmakers E.-J., Ware J.J., Ioannidis J.P.A. A manifesto for reproducible science. Nat. Hum. Behav. 2017;1:0021. PubMed PMC

Hutson M. Artificial intelligence faces reproducibility crisis. Science. 2018;359:725–726. PubMed

Ellenberg J., Swedlow J.R., Barlow M., Cook C.E., Sarkans U., Patwardhan A., Brazma A., Birney E. A call for public archives for biological image data. Nat. Methods. 2018;15:849–854. PubMed PMC

Allan C., Burel J.M., Moore J., Blackburn C., Linkert M., Loynton S., Macdonald D., Moore W.J., Neves C., Patterson A. OMERO: flexible, model-driven data management for experimental biology. Nat. Methods. 2012;9:245–253. PubMed PMC

Kvilekval K., Fedorov D., Obara B., Singh A., Manjunath B.S. Bisque: a platform for bioimage analysis and management. Bioinformatics. 2010;26:544–552. PubMed

Williams E., Moore J., Li S.W., Rustici G., Tarkowska A., Chessel A., Leo S., Antal B., Ferguson R.K., Sarkans U. Image Data Resource: a bioimage data integration and publication platform. Nat. Methods. 2017;14:775–781. PubMed PMC

Vandewalle P. Code sharing is associated with research impact in image processing. Comput. Sci. Eng. 2012;14:42–47.

Maier-Hein L., Eisenmann M., Reinke A., Onogur S., Stankovic M., Scholz P., Arbel T., Bogunovic H., Bradley A.P., Carass A. Why rankings of biomedical image analysis competitions should be interpreted with care. Nat. Commun. 2018;9:5217. PubMed PMC

Meijering E., Carpenter A., Peng H., Hamprecht F.A., Olivo-Marin J.C. Imagining the future of bioimage analysis. Nat. Biotechnol. 2016;34:1250–1255. PubMed

Perkel J.M. A toolkit for data transparency takes shape. Nature. 2018;560:513–515. PubMed

Grüning B.A., Rasche E., Rebolledo-Jaramillo B., Eberhard C., Houwaart T., Chilton J., Coraor N., Backofen R., Taylor J., Nekrutenkoet A. Jupyter and Galaxy: easing entry barriers into complex data analyses for biomedical researchers. PLoS Comput. Biol. 2017;13:e1005425. PubMed PMC

Marée R., Rollus L., Stévens B., Hoyoux R., Louppe G., Vandaele R., Begon J.M., Kainz P., Geurts P., Wehenkel L. Collaborative analysis of multi-gigapixel imaging data with Cytomine. Bioinformatics. 2016;32:1395–1401. PubMed PMC

Glatard T., Kiar G., Aumentado-Armstrong T., Beck N., Bellec P., Bernard R., Bonnet A., Brown S.T., Camarasu-Pop S., Cervenansky F. Boutiques: a flexible framework to integrate command-line applications in computing platforms. GigaScience. 2018;7:giy016. PubMed PMC

Kurtzer G.M., Sochat V., Bauer M.W. Singularity: scientific containers for mobility of compute. PLoS One. 2017;12:e0177459. PubMed PMC

Yoo A., Jette M., Grondona M. SLURM: simple Linux utility for resource management, job scheduling strategies for parallel processing. Lect. Notes Comput. Sci. 2003;2862:44–60.

Kozubek M. Challenges and benchmarks in bioimage analysis. Adv. Anat. Embryol. Cell Biol. 2016;219:231–262. PubMed

Brown K.M., Barrionuevo G., Canty A.J., De Paola V., Hirsch J.A., Jefferis G.S., Lu J., Snippe M., Sugihara I., Ascoli G.A. The DIADEM data sets: representative light microscopy images of neuronal morphology to advance automation of digital reconstructions. Neuroinformatics. 2011;9:143–157. PubMed PMC

Ulman V., Maška M., Magnusson K.E.G., Ronneberger O., Haubold C., Harder N., Matula P., Matula P., Svoboda D., Radojevic M. An objective comparison of cell-tracking algorithms. Nat. Methods. 2017;14:1141–1152. PubMed PMC

Chenouard N., Smal I., de Chaumont F., Maška M., Sbalzarini I.F., Gong Y., Cardinale J., Carthel C., Coraluppi S., Winter M. Objective comparison of particle tracking methods. Nat. Methods. 2014;11:281–289. PubMed PMC

Caicedo J.C., Goodman A., Karhohs K.W., Cimini B.A., Ackerman J., Haghighi M., Heng C., Becker T., Doan M., McQuin C. Nucleus segmentation across imaging experiments: the 2018 data science bowl. Nat. Methods. 2019;16:1247–1253. PubMed PMC

Svoboda D., Kozubek M., Stejskal S. Generation of digital phantoms of cell nuclei and simulation of image formation in 3D image cytometry. Cytometry A. 2009;75:494–509. PubMed

Wiesner D., Svoboda D., Maška M., Kozubek M. CytoPacq: a web-interface for simulating multi-dimensional cell imaging. Bioinformatics. 2019;35:4531–4533. PubMed PMC

Cuntz H., Forstner F., Borst A., Häusser M. One rule to grow them all: a general theory of neuronal branching and its practical application. PLoS Comput. Biol. 2010;6:e1000877. PubMed PMC

Jassi P., Hamarneh G. VascuSynth: vascular tree synthesis software. Insight J. 2011 http://hdl.handle.net/10380/3260

Lehmussola A., Ruusuvuori P., Selinummi J., Huttunen H., Yli-Harja O. Computational framework for simulating fluorescence microscope images with cell populations. IEEE Trans. Med. Imaging. 2007;26:1010–1016. PubMed

Vandaele R., Aceto J., Muller M., Péronnet F., Debat V., Wang C.W., Huang C.T., Jodogne S., Martinive P., Geurts P. Landmark detection in 2D bioimages for geometric morphometrics: a multi-resolution tree-based approach. Sci. Rep. 2018;8:538. PubMed PMC

Schneider C.A., Rasband W.S., Eliceiri K.W. NIH Image to ImageJ: 25 years of image analysis. Nat. Methods. 2012;9:671–675. PubMed PMC

de Chaumont F., Dallongeville S., Chenouard N., Hervé N., Pop S., Provoost T., Meas-Yedid V., Pankajakshan P., Lecomte T., Le Montagner Y. Icy: an open bioimage informatics platform for extended reproducible research. Nat. Methods. 2012;9:690–696. PubMed

McQuin C., Goodman A., Chernyshev V., Kamentsky L., Cimini B.A., Karhohs K.W., Doan M., Ding L., Rafelski S.M., Thirstrup D. CellProfiler 3.0: next-generation image processing for biology. PLoS Biol. 2018;16:e2005970. PubMed PMC

Peng H., Ruan Z., Long F., Simpson J.H., Myers E.W. V3D enables real-time 3D visualization and quantitative analysis of large-scale biological image data sets. Nat. Biotechnol. 2010;28:348–353. PubMed PMC

Berg S., Kutra D., Kroeger T., Straehle C.N., Kausler B.X., Haubold C., Schiegg M., Ales J., Beier T., Rudy M. ilastik: interactive machine learning for (bio)image analysis. Nat. Methods. 2019;16:1226–1232. PubMed

Eaton J.W., Bateman D., Hauberg S., Wehbring R. Free Software Foundation; 2016. GNU Octave Version 4.2.0 Manual: A High-Level Interactive Language for Numerical Computations.https://octave.org/doc/octave-4.2.0.pdf

Pedregosa F., Varoquaux G., Gramfort A., Michel V., Thirion B., Grisel O., Blondel M., Prettenhofer P., Weiss R., Dubourg V. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 2011;12:2825–2830.

Chollet F. Manning; 2017. Deep Learning with Python.

Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A. (2017). Automatic Differentiation in PyTorch. NIPS Autodiff Workshop, 2017.

Sirinukunwattana K., Pluim J.P.W., Chen H., Qi X., Heng P.A., Guo Y.B., Wang L.Y., Matuszewski B.J., Bruni E., Sanchez U. Gland segmentation in colon histology images: the glas challenge contest. Med. Image Anal. 2017;35:489–502. PubMed

Multon S., Pesesse L., Weatherspoon A., Florquin S., Van de Poel J.F., Martin P., Vincke G., Hoyoux R., Marée R., Verpoorten D. A Massive Open Online Course (MOOC) on practical histology: a goal, a tool, a large public! Return on a first experience. Ann. Pathol. 2018;38:76–84. PubMed

Ellenberg J., Swedlow J.R., Barlow M., Cook C.E., Sarkans U., Patwardhan A., Brazma A., Birney E. A call for public archives for biological image data. Nat. Methods. 2018;15:849–854. PubMed PMC