Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

• MeSH
• umělá inteligence * MeSH
• Publikační typ
• časopisecké články MeSH
• přehledy MeSH

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

• MeSH
• algoritmy * MeSH
• počítačové zpracování obrazu * MeSH
• sémantika MeSH
• strojové učení MeSH
• Publikační typ
• časopisecké články MeSH
• přehledy MeSH

BACKGROUND: Structural cortical networks (SCNs) represent patterns of coordinated morphological modifications in cortical areas, and they present the advantage of being extracted from previously acquired clinical magnetic resonance imaging (MRI) scans. SCNs have shown pathophysiological changes in many brain disorders, including multiple sclerosis. OBJECTIVE: To investigate alterations of SCNs at the individual level in patients with clinically isolated syndrome (CIS), thereby assessing their clinical relevance. METHODS: We analyzed baseline data collected in a prospective multicenter (MAGNIMS) study. CIS patients (n = 60) and healthy controls (n = 38) underwent high-resolution 3T MRI. Measures of disability and cognitive processing were obtained for patients. Single-subject SCNs were extracted from brain 3D-T1 weighted sequences; global and local network parameters were computed. RESULTS: Compared to healthy controls, CIS patients showed altered small-world topology, an efficient network organization combining dense local clustering with relatively few long-distance connections. These disruptions were worse for patients with higher lesion load and worse cognitive processing speed. Alterations of centrality measures and clustering of connections were observed in specific cortical areas in CIS patients when compared with healthy controls. CONCLUSION: Our study indicates that SCNs can be used to demonstrate clinically relevant alterations of connectivity in CIS.

• MeSH
• demyelinizační nemoci * diagnostické zobrazování   MeSH
• kognice MeSH
• lidé MeSH
• magnetická rezonanční tomografie MeSH
• mozek diagnostické zobrazování   MeSH
• nervové dráhy diagnostické zobrazování   MeSH
• prospektivní studie MeSH
• Check Tag
• lidé MeSH
• Publikační typ
• časopisecké články MeSH
• multicentrická studie MeSH
• práce podpořená grantem MeSH