Detail
Article
Online article
FT
Medvik - BMC
  • Something wrong with this record ?

Why rankings of biomedical image analysis competitions should be interpreted with care

L. Maier-Hein, M. Eisenmann, A. Reinke, S. Onogur, M. Stankovic, P. Scholz, T. Arbel, H. Bogunovic, AP. Bradley, A. Carass, C. Feldmann, AF. Frangi, PM. Full, B. van Ginneken, A. Hanbury, K. Honauer, M. Kozubek, BA. Landman, K. März, O. Maier, K....

. 2018 ; 9 (1) : 5217. [pub] 20181206

Language English Country England, Great Britain

Document type Journal Article, Research Support, N.I.H., Extramural, Research Support, Non-U.S. Gov't, Research Support, U.S. Gov't, Non-P.H.S.

Persistent link   https://www.medvik.cz/link/bmc19012053

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

AIExplore NTUST Center of Computer Vision and Medical Imaging Graduate Institute of Biomedical Engineering National Taiwan University of Science and Technology Taipei 106 Taiwan

Centre for Biomedical Image Analysis Masaryk University 60200 Brno Czech Republic

Centre for Intelligent Machines McGill University Montreal QC H3A0G4 Canada

Centre for Medical Image Computing and Department of Computer Science University College London London W1W 7TS UK

Christian Doppler Laboratory for Ophthalmic Image Analysis Department of Ophthalmology Medical University Vienna 1090 Vienna Austria

CISTIB Center for Computational Imaging and Simulation Technologies in Biomedicine The University of Leeds Leeds Yorkshire LS2 9JT UK

Data Science Studio Research Studios Austria FG 1090 Vienna Austria

Department of Computer Science University of Warwick Coventry CV4 7AL UK

Department of Electrical and Computer Engineering Department of Computer Science Johns Hopkins University Baltimore MD 21218 USA

Department of Electrical Engineering Eindhoven University of Technology 5600 MB Eindhoven The Netherlands

Department of Radiation Oncology Massachusetts General Hospital Boston MA 02114 USA

Department of Radiology and Nuclear Medicine Medical Image Analysis Radboud University Center 6525 GA Nijmegen The Netherlands

Departments of Radiology Nuclear Medicine and Medical Informatics Erasmus MC 3015 GD Rotterdam The Netherlands

Division of Biostatistics German Cancer Research Center 69120 Heidelberg Germany

Division of Clinical Epidemiology and Aging Research German Cancer Research Center 69120 Heidelberg Germany

Division of Computer Assisted Medical Interventions 69120 Heidelberg Germany

Division of Medical Image Computing 69120 Heidelberg Germany

Division of Translational Surgical Oncology National Center for Tumor Diseases Dresden 01307 Dresden Germany

Electrical Engineering Vanderbilt University Nashville TN 37235 1679 USA

Heidelberg Collaboratory for Image Processing Heidelberg University 69120 Heidelberg Germany

Information System Institute HES SO Sierre 3960 Switzerland

Institute for Advanced Studies Department of Informatics Technical University of Munich 80333 Munich Germany

Institute for Surgical Technology and Biomechanics University of Bern Bern 3014 Switzerland

Institute of Biomedical Engineering University of Oxford Oxford OX3 7DQ UK

Institute of Diagnostic and Interventional Radiology University Medical Center Rostock 18051 Rostock Germany

Institute of Information Systems Engineering TU Wien 1040 Vienna Austria Complexity Science Hub Vienna 1080 Vienna Austria

Institute of Medical Informatics Universität zu Lübeck 23562 Lübeck Germany

Science and Engineering Faculty Queensland University of Technology Brisbane QLD 4001 Australia

Univ Rennes Inserm LTSI UMR_S 1099 Rennes 35043 Cedex France

References provided by Crossref.org

000      
00000naa a2200000 a 4500
001      
bmc19012053
003      
CZ-PrNML
005      
20190405092820.0
007      
ta
008      
190405s2018 enk f 000 0|eng||
009      
AR
024    7_
$a 10.1038/s41467-018-07619-7 $2 doi
035    __
$a (PubMed)30523263
040    __
$a ABA008 $b cze $d ABA008 $e AACR2
041    0_
$a eng
044    __
$a enk
100    1_
$a Maier-Hein, Lena $u Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany. l.maier-hein@dkfz.de.
245    10
$a Why rankings of biomedical image analysis competitions should be interpreted with care / $c L. Maier-Hein, M. Eisenmann, A. Reinke, S. Onogur, M. Stankovic, P. Scholz, T. Arbel, H. Bogunovic, AP. Bradley, A. Carass, C. Feldmann, AF. Frangi, PM. Full, B. van Ginneken, A. Hanbury, K. Honauer, M. Kozubek, BA. Landman, K. März, O. Maier, K. Maier-Hein, BH. Menze, H. Müller, PF. Neher, W. Niessen, N. Rajpoot, GC. Sharp, K. Sirinukunwattana, S. Speidel, C. Stock, D. Stoyanov, AA. Taha, F. van der Sommen, CW. Wang, MA. Weber, G. Zheng, P. Jannin, A. Kopp-Schneider,
520    9_
$a International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.
650    _2
$a biomedicínský výzkum $x metody $x normy $7 D035843
650    _2
$a biomedicínské technologie $x klasifikace $x metody $x normy $7 D020811
650    _2
$a diagnostické zobrazování $x klasifikace $x metody $x normy $7 D003952
650    _2
$a lidé $7 D006801
650    _2
$a počítačové zpracování obrazu $x metody $x normy $7 D007091
650    _2
$a reprodukovatelnost výsledků $7 D015203
650    _2
$a průzkumy a dotazníky $7 D011795
650    _2
$a hodnocení biomedicínských technologií $x metody $x normy $7 D013673
655    _2
$a časopisecké články $7 D016428
655    _2
$a Research Support, N.I.H., Extramural $7 D052061
655    _2
$a práce podpořená grantem $7 D013485
655    _2
$a Research Support, U.S. Gov't, Non-P.H.S. $7 D013486
700    1_
$a Eisenmann, Matthias $u Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
700    1_
$a Reinke, Annika $u Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
700    1_
$a Onogur, Sinan $u Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
700    1_
$a Stankovic, Marko $u Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
700    1_
$a Scholz, Patrick $u Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
700    1_
$a Arbel, Tal $u Centre for Intelligent Machines, McGill University, Montreal, QC, H3A0G4, Canada.
700    1_
$a Bogunovic, Hrvoje $u Christian Doppler Laboratory for Ophthalmic Image Analysis, Department of Ophthalmology, Medical University Vienna, 1090, Vienna, Austria.
700    1_
$a Bradley, Andrew P $u Science and Engineering Faculty, Queensland University of Technology, Brisbane, QLD, 4001, Australia.
700    1_
$a Carass, Aaron $u Department of Electrical and Computer Engineering, Department of Computer Science, Johns Hopkins University, Baltimore, MD, 21218, USA.
700    1_
$a Feldmann, Carolin $u Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
700    1_
$a Frangi, Alejandro F $u CISTIB - Center for Computational Imaging & Simulation Technologies in Biomedicine, The University of Leeds, Leeds, Yorkshire, LS2 9JT, UK.
700    1_
$a Full, Peter M $u Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
700    1_
$a van Ginneken, Bram $u Department of Radiology and Nuclear Medicine, Medical Image Analysis, Radboud University Center, 6525 GA, Nijmegen, The Netherlands.
700    1_
$a Hanbury, Allan $u Institute of Information Systems Engineering, TU Wien, 1040, Vienna, Austria. Complexity Science Hub Vienna, 1080, Vienna, Austria.
700    1_
$a Honauer, Katrin $u Heidelberg Collaboratory for Image Processing (HCI), Heidelberg University, 69120, Heidelberg, Germany.
700    1_
$a Kozubek, Michal $u Centre for Biomedical Image Analysis, Masaryk University, 60200, Brno, Czech Republic.
700    1_
$a Landman, Bennett A $u Electrical Engineering, Vanderbilt University, Nashville, TN, 37235-1679, USA.
700    1_
$a März, Keno $u Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
700    1_
$a Maier, Oskar $u Institute of Medical Informatics, Universität zu Lübeck, 23562, Lübeck, Germany.
700    1_
$a Maier-Hein, Klaus $u Division of Medical Image Computing (MIC), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
700    1_
$a Menze, Bjoern H $u Institute for Advanced Studies, Department of Informatics, Technical University of Munich, 80333, Munich, Germany.
700    1_
$a Müller, Henning $u Information System Institute, HES-SO, Sierre, 3960, Switzerland.
700    1_
$a Neher, Peter F $u Division of Medical Image Computing (MIC), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
700    1_
$a Niessen, Wiro $u Departments of Radiology, Nuclear Medicine and Medical Informatics, Erasmus MC, 3015 GD, Rotterdam, The Netherlands.
700    1_
$a Rajpoot, Nasir $u Department of Computer Science, University of Warwick, Coventry, CV4 7AL, UK.
700    1_
$a Sharp, Gregory C $u Department of Radiation Oncology, Massachusetts General Hospital, Boston, MA, 02114, USA.
700    1_
$a Sirinukunwattana, Korsuk $u Institute of Biomedical Engineering, University of Oxford, Oxford, OX3 7DQ, UK.
700    1_
$a Speidel, Stefanie $u Division of Translational Surgical Oncology (TCO), National Center for Tumor Diseases Dresden, 01307, Dresden, Germany.
700    1_
$a Stock, Christian $u Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
700    1_
$a Stoyanov, Danail $u Centre for Medical Image Computing (CMIC) & Department of Computer Science, University College London, London, W1W 7TS, UK.
700    1_
$a Taha, Abdel Aziz $u Data Science Studio, Research Studios Austria FG, 1090, Vienna, Austria.
700    1_
$a van der Sommen, Fons $u Department of Electrical Engineering, Eindhoven University of Technology, 5600 MB, Eindhoven, The Netherlands.
700    1_
$a Wang, Ching-Wei $u AIExplore, NTUST Center of Computer Vision and Medical Imaging, Graduate Institute of Biomedical Engineering, National Taiwan University of Science and Technology, Taipei, 106, Taiwan.
700    1_
$a Weber, Marc-André $u Institute of Diagnostic and Interventional Radiology, University Medical Center Rostock, 18051, Rostock, Germany.
700    1_
$a Zheng, Guoyan $u Institute for Surgical Technology and Biomechanics, University of Bern, Bern, 3014, Switzerland.
700    1_
$a Jannin, Pierre $u Univ Rennes, Inserm, LTSI (Laboratoire Traitement du Signal et de l'Image) - UMR_S 1099, Rennes, 35043, Cedex, France.
700    1_
$a Kopp-Schneider, Annette $u Division of Biostatistics, German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
773    0_
$w MED00184850 $t Nature communications $x 2041-1723 $g Roč. 9, č. 1 (2018), s. 5217
856    41
$u https://pubmed.ncbi.nlm.nih.gov/30523263 $y Pubmed
910    __
$a ABA008 $b sig $c sign $y a $z 0
990    __
$a 20190405 $b ABA008
991    __
$a 20190405092829 $b ABA008
999    __
$a ok $b bmc $g 1391363 $s 1050358
BAS    __
$a 3
BAS    __
$a PreBMC
BMC    __
$a 2018 $b 9 $c 1 $d 5217 $e 20181206 $i 2041-1723 $m Nature communications $n Nat Commun $x MED00184850
LZP    __
$a Pubmed-20190405
Back

Find record

Citation metrics

Archiving options