Benchmarks Dotaz Zobrazit nápovědu
Similar to the medical imaging community, the bioimaging community has recently realized the need to benchmark various image analysis methods to compare their performance and assess their suitability for specific applications. Challenges sponsored by prestigious conferences have proven to be an effective means of encouraging benchmarking and new algorithm development for a particular type of image data. Bioimage analysis challenges have recently complemented medical image analysis challenges, especially in the case of the International Symposium on Biomedical Imaging (ISBI). This review summarizes recent progress in this respect and describes the general process of designing a bioimage analysis benchmark or challenge, including the proper selection of datasets and evaluation metrics. It also presents examples of specific target applications and biological research tasks that have benefited from these challenges with respect to the performance of automatic image analysis methods that are crucial for the given task. Finally, available benchmarks and challenges in terms of common features, possible classification and implications drawn from the results are analysed.
- MeSH
- algoritmy MeSH
- benchmarking * MeSH
- databáze faktografické MeSH
- fluorescenční mikroskopie přístrojové vybavení metody normy MeSH
- lidé MeSH
- molekulární zobrazování přístrojové vybavení metody normy MeSH
- počítačové zpracování obrazu metody statistika a číselné údaje MeSH
- rozpoznávání automatizované statistika a číselné údaje MeSH
- Check Tag
- lidé MeSH
- Publikační typ
- časopisecké články MeSH
- práce podpořená grantem MeSH
- přehledy MeSH
BACKGROUND: Recently, deep neural networks have been successfully applied in many biological fields. In 2020, a deep learning model AlphaFold won the protein folding competition with predicted structures within the error tolerance of experimental methods. However, this solution to the most prominent bioinformatic challenge of the past 50 years has been possible only thanks to a carefully curated benchmark of experimentally predicted protein structures. In Genomics, we have similar challenges (annotation of genomes and identification of functional elements) but currently, we lack benchmarks similar to protein folding competition. RESULTS: Here we present a collection of curated and easily accessible sequence classification datasets in the field of genomics. The proposed collection is based on a combination of novel datasets constructed from the mining of publicly available databases and existing datasets obtained from published articles. The collection currently contains nine datasets that focus on regulatory elements (promoters, enhancers, open chromatin region) from three model organisms: human, mouse, and roundworm. A simple convolution neural network is also included in a repository and can be used as a baseline model. Benchmarks and the baseline model are distributed as the Python package 'genomic-benchmarks', and the code is available at https://github.com/ML-Bioinfo-CEITEC/genomic_benchmarks . CONCLUSIONS: Deep learning techniques revolutionized many biological fields but mainly thanks to the carefully curated benchmarks. For the field of Genomics, we propose a collection of benchmark datasets for the classification of genomic sequences with an interface for the most commonly used deep learning libraries, implementation of the simple neural network and a training framework that can be used as a starting point for future research. The main aim of this effort is to create a repository for shared datasets that will make machine learning for genomics more comparable and reproducible while reducing the overhead of researchers who want to enter the field, leading to healthy competition and new discoveries.
- Klíčová slova
- Benchmark, Convolutional neural network, Dataset, Deep learning, Genomics,
- MeSH
- benchmarking * MeSH
- chromatin MeSH
- genomika metody MeSH
- lidé MeSH
- myši MeSH
- neuronové sítě (počítačové) * MeSH
- strojové učení MeSH
- zvířata MeSH
- Check Tag
- lidé MeSH
- myši MeSH
- zvířata MeSH
- Publikační typ
- časopisecké články MeSH
- práce podpořená grantem MeSH
- Názvy látek
- chromatin MeSH
Forests are increasingly affected by natural disturbances. Subsequent salvage logging, a widespread management practice conducted predominantly to recover economic capital, produces further disturbance and impacts biodiversity worldwide. Hence, naturally disturbed forests are among the most threatened habitats in the world, with consequences for their associated biodiversity. However, there are no evidence-based benchmarks for the proportion of area of naturally disturbed forests to be excluded from salvage logging to conserve biodiversity. We apply a mixed rarefaction/extrapolation approach to a global multi-taxa dataset from disturbed forests, including birds, plants, insects and fungi, to close this gap. We find that 75 ± 7% (mean ± SD) of a naturally disturbed area of a forest needs to be left unlogged to maintain 90% richness of its unique species, whereas retaining 50% of a naturally disturbed forest unlogged maintains 73 ± 12% of its unique species richness. These values do not change with the time elapsed since disturbance but vary considerably among taxonomic groups.
- MeSH
- benchmarking MeSH
- biodiverzita MeSH
- druhová specificita MeSH
- ekosystém MeSH
- lesnictví normy MeSH
- lesy * MeSH
- zachování přírodních zdrojů * metody MeSH
- zvířata MeSH
- Check Tag
- zvířata MeSH
- Publikační typ
- časopisecké články MeSH
- práce podpořená grantem MeSH
The aim was to provide benchmarks and investigate contribution of start and turn performances in the European Short-Course Swimming Championship. Over all race distances, 932 individual races of male competitors were video captured and the start and turn performances were analysed. Start and turn performances contributed up to 26% and 56% of the total race time. Analysis of variance showed that the 15 m start times were faster for Freestyle and Butterfly (p < 0.001) compared to the other swimming strokes. In-water starts (Backstroke) were slower at the 5 m mark compare to on-block starts (p < 0.001). Tumble turns were faster than open turns (p < 0.001). Multiple linear regression analysis explained 82-97% of total variance in the race results with a decreasing effect of start performance for the longer distance races. Turn performance affected the results across all race distances (p < 0.001). The benchmarks and percentiles provide comparative data for swimmers of different performance levels. Considering the large contribution of start and turn performance to race time and the high effect in the regression model, training regimes that are mainly based on conditioning of free-swimming should be reconsidered.
- Klíčová slova
- Athlete, competition, key performance indicators, performance analysis, swimming,
- MeSH
- benchmarking * MeSH
- biomechanika MeSH
- kompetitivní chování MeSH
- lidé MeSH
- plavání MeSH
- sportovní výkon * MeSH
- Check Tag
- lidé MeSH
- mužské pohlaví MeSH
- Publikační typ
- časopisecké články MeSH
BACKGROUND: Benchmark comparisons in surgery allow identification of gaps in the quality of care provided. The aim of this study was to determine quality thresholds for high (HAR) and low (LAR) anterior resections in colorectal cancer surgery by applying the concept of benchmarking. METHODS: This 5-year multinational retrospective study included patients who underwent anterior resection for cancer in 19 high-volume centres on five continents. Benchmarks were defined for 11 relevant postoperative variables at discharge, 3 months, and 6 months (for LAR). Benchmarks were calculated for two separate cohorts: patients without (ideal) and those with (non-ideal) outcome-relevant co-morbidities. Benchmark cut-offs were defined as the 75th percentile of each centre's median value. RESULTS: A total of 3903 patients who underwent HAR and 3726 who had LAR for cancer were analysed. After 3 months' follow-up, the mortality benchmark in HAR for ideal and non-ideal patients was 0.0 versus 3.0 per cent, and in LAR it was 0.0 versus 2.2 per cent. Benchmark results for anastomotic leakage were 5.0 versus 6.9 per cent for HAR, and 13.6 versus 11.8 per cent for LAR. The overall morbidity benchmark in HAR was a Comprehensive Complication Index (CCI®) score of 8.6 versus 14.7, and that for LAR was CCI® score 11.9 versus 18.3. CONCLUSION: Regular comparison of individual-surgeon or -unit outcome data against benchmark thresholds may identify gaps in care quality that can improve patient outcome.
- MeSH
- benchmarking MeSH
- kolorektální chirurgie * MeSH
- lidé MeSH
- nádory rekta * chirurgie MeSH
- pooperační komplikace epidemiologie etiologie MeSH
- proktektomie * MeSH
- retrospektivní studie MeSH
- Check Tag
- lidé MeSH
- Publikační typ
- časopisecké články MeSH
Interpretation of QSAR models is useful to understand the complex nature of biological or physicochemical processes, guide structural optimization or perform knowledge-based validation of QSAR models. Highly predictive models are usually complex and their interpretation is non-trivial. This is particularly true for modern neural networks. Various approaches to interpretation of these models exist. However, it is difficult to evaluate and compare performance and applicability of these ever-emerging methods. Herein, we developed several benchmark data sets with end-points determined by pre-defined patterns. These data sets are purposed for evaluation of the ability of interpretation approaches to retrieve these patterns. They represent tasks with different complexity levels: from simple atom-based additive properties to pharmacophore hypothesis. We proposed several quantitative metrics of interpretation performance. Applicability of benchmarks and metrics was demonstrated on a set of conventional models and end-to-end graph convolutional neural networks, interpreted by the previously suggested universal ML-agnostic approach for structural interpretation. We anticipate these benchmarks to be useful in evaluation of new interpretation approaches and investigation of decision making of complex "black box" models.
- Klíčová slova
- Atom contributions, Benchmark data set, Graph convolutional neural networks, Interpretability metrics, QSAR model interpretation, Synthetic data set,
- Publikační typ
- časopisecké články MeSH
MOTIVATION: Automatic tracking of cells in multidimensional time-lapse fluorescence microscopy is an important task in many biomedical applications. A novel framework for objective evaluation of cell tracking algorithms has been established under the auspices of the IEEE International Symposium on Biomedical Imaging 2013 Cell Tracking Challenge. In this article, we present the logistics, datasets, methods and results of the challenge and lay down the principles for future uses of this benchmark. RESULTS: The main contributions of the challenge include the creation of a comprehensive video dataset repository and the definition of objective measures for comparison and ranking of the algorithms. With this benchmark, six algorithms covering a variety of segmentation and tracking paradigms have been compared and ranked based on their performance on both synthetic and real datasets. Given the diversity of the datasets, we do not declare a single winner of the challenge. Instead, we present and discuss the results for each individual dataset separately. AVAILABILITY AND IMPLEMENTATION: The challenge Web site (http://www.codesolorzano.com/celltrackingchallenge) provides access to the training and competition datasets, along with the ground truth of the training videos. It also provides access to Windows and Linux executable files of the evaluation software and most of the algorithms that competed in the challenge.
AIM: Despite the existence of evidence-based guidelines for the care of children with diabetes, widespread gaps in knowledge, attitude, and practice remain. The purpose of this paper is to present a review of benchmarking practices and results of this process within SWEET, moreover focusing on current challenges and future directions. METHODS: Biannually, members electronically transfer de-identified clinic data for 37 parameters to the SWEET database. Each center receives benchmarking and data validation reports. RESULTS: In 2015, 48 centers have contributed data for 20 165 unique patients (51.6% male). After exclusion for missing data 19 131 patients remain for further analysis. The median age is 14.2 years, with a median diabetes duration 4.8 years; 96.0% of patients have type 1, 1.1% type 2, and 2.9% other diabetes types. Data completeness has increased over time. In 2015, median HbA1c of all patients' (diabetes type 1) medians was 7.8% (61.7 mmol/mol) with 39.1%, 41.4%, and 19.4% of patients having HbA1c < 7.5% (58 mmol/mol), 7.5%-9% (58-75 mmol/mol) and >9% (75 mmol/mol), respectively. Although HbA1c has been stable over time [7.7%-7.8% (60.7-61.7 mmol/mol)], there remains wide variation between centers. Fourteen centers achieve a median HbA1c <7.5% (58 mmol/mol). CONCLUSIONS: Our vision is that the participation in SWEET is encouraging members to deliver increasingly accurate and complete data. Dissemination of results and prospective projects serve as further motivation to improve data reporting. Comparing processes and outcomes will help members identify weaknesses and introduce innovative solutions, resulting in improved and more uniform care for patients with diabetes.
- Klíčová slova
- benchmarking, diabetes, international comparison, pediatric,
- MeSH
- benchmarking * MeSH
- diabetes mellitus epidemiologie terapie MeSH
- dítě MeSH
- lidé MeSH
- mladiství MeSH
- pediatrie normy MeSH
- Check Tag
- dítě MeSH
- lidé MeSH
- mladiství MeSH
- mužské pohlaví MeSH
- ženské pohlaví MeSH
- Publikační typ
- časopisecké články MeSH
- multicentrická studie MeSH
- práce podpořená grantem MeSH
- přehledy MeSH
Computational models of acoustic wave propagation are frequently used in transcranial ultrasound therapy, for example, to calculate the intracranial pressure field or to calculate phase delays to correct for skull distortions. To allow intercomparison between the different modeling tools and techniques used by the community, an international working group was convened to formulate a set of numerical benchmarks. Here, these benchmarks are presented, along with intercomparison results. Nine different benchmarks of increasing geometric complexity are defined. These include a single-layer planar bone immersed in water, a multi-layer bone, and a whole skull. Two transducer configurations are considered (a focused bowl and a plane piston operating at 500 kHz), giving a total of 18 permutations of the benchmarks. Eleven different modeling tools are used to compute the benchmark results. The models span a wide range of numerical techniques, including the finite-difference time-domain method, angular spectrum method, pseudospectral method, boundary-element method, and spectral-element method. Good agreement is found between the models, particularly for the position, size, and magnitude of the acoustic focus within the skull. When comparing results for each model with every other model in a cross-comparison, the median values for each benchmark for the difference in focal pressure and position are less than 10% and 1 mm, respectively. The benchmark definitions, model results, and intercomparison codes are freely available to facilitate further comparisons.
In this critical review, we comment on the absence of widely shared benchmark problems and relevant challenges or even attractive competitions in the field of electrokinetics. We argue that in some other scientific domains that are, similarly as electrokinetics, strongly multidisciplinary, the existence of these tools is very beneficial because it stimulates the discussion about what constitutes the bottleneck of further progress, allows easier exploitation of results provided by other scientific and engineering disciplines, and, last but not least, makes the research domain attractive and visible to a broader public, including students. The goal of this review is to provoke some discussion that might perhaps lead to compensating for these shortcomings.
- Klíčová slova
- Benchmark, Challenge, Competition, Dielectrophoresis, Electrokinetics,
- MeSH
- benchmarking MeSH
- elektroforéza * metody normy MeSH
- počítačová simulace MeSH
- robotika MeSH
- výzkum MeSH
- Publikační typ
- časopisecké články MeSH
- práce podpořená grantem MeSH
- přehledy MeSH
