receiver operating curve
Dotaz
Zobrazit nápovědu
Ve své práci jsme předložili přehled o ROC (receiver operating characteristic) analýze a jejím použití v medicíně. Článek uvádí krátký přehled teorie i způsob, jak lze ROC křivku vytvořit, a dále zdůrazňuje význam analýzy nákladů a přínosů (Cost-Benefit Analysis) při volbě optimálního dělícího bodu (prahu). Použití ROC analýzy jsme ukázali na několika příkladech v části „Analýza nákladů a přínosů“. Na těchto příkladech vidíme, že pro určení optimálního dělícího bodu má rozhodující význam prevalence onemocnění, závažnost onemocnění, rizika a nežádoucí účinky léčby nebo diagnostického testu, celkové náklady na léčbu pravdivě i falešně pozitivních pacientů i riziko nedostatečné nebo žádné léčby u falešně negativních.
An overview of the use of Receiver Operating Characteristic (ROC) analysis within medicine is provided. A survey of the theory behind the analysis is offered together with a presentation on how to create a ROC curve and how to use Cost – Benefit analysis to determine the optimal cut-off point or threshold. The use of ROC analysis is exemplified in the “Cost – Benefit analysis” section of the paper. In these examples, it can be seen that the determination of the optimal cut-off point is mainly influenced by the prevalence and the severity of the disease, by the risks and adverse events of treatment or the diagnostic testing, by the overall costs of treating true and false positives (TP and FP), and by the risk of deficient or non-treatment of false negative (FN) cases.
BACKGROUND: Although fall prevention in patients after stroke is crucial, the clinical validity of fall risk assessment tools is underresearched in this population. The study aim was to determine the cut-off scores and clinical validity of the Sensory Organization Test (SOT), the Berg Balance Scale (BBS), and the Fall Efficacy Scale-International (FES-I) in patients after stroke. METHODS: In this prospective cross-sectional study, we analyzed data for patients admitted to a rehabilitation unit after stroke from 2018 through 2021. Participants underwent SOT, BBS, and FES-I pre-discharge, and the fall incidence was recorded for 6 months. We used an area under the receiver operating characteristic curve (AUC) to calculate predictive values. RESULTS: Of 84 included patients (median age 68.5 (interquartile range 67-71) years), 32 (38.1%) suffered a fall. All three tests were significantly predictive of fall risk. Optimal cut-off scores were 60 points for SOT (AUC 0.686), 35 and 42 points for BBS (AUC 0.661 and 0.618, respectively), and 27 and 29 points for FES-I (AUC 0.685 and 0.677, respectively). CONCLUSIONS: Optimal cut-off scores for SOT, BBS, and FES-I were determined for patients at risk for falls after a stroke, which all three tools classified with a good discriminatory ability.
BACKGROUND: The heterogeneity and lack of validation of existing severity scores for food allergic reactions limit standardization of case management and research advances. We aimed to develop and validate a severity score for food allergic reactions. METHODS: Following a multidisciplinary experts consensus, it was decided to develop a food allergy severity score (FASS) with ordinal (oFASS) and numerical (nFASS) formats. oFASS with 3 and 5 grades were generated through expert consensus, and nFASS by mathematical modeling. Evaluation was performed in the EuroPrevall outpatient clinic cohort (8232 food reactions) by logistic regression with request of emergency care and medications used as outcomes. Discrimination, classification, and calibration were calculated. Bootstrapping internal validation was followed by external validation (logistic regression) in 5 cohorts (3622 food reactions). Correlation of nFASS with the severity classification done by expert allergy clinicians by Best-Worst Scaling of 32 food reactions was calculated. RESULTS: oFASS and nFASS map consistently, with nFASS having greater granularity. With the outcomes emergency care, adrenaline and critical medical treatment, oFASS and nFASS had a good discrimination (receiver operating characteristic area under the curve [ROC-AUC]>0.80), classification (sensitivity 0.87-0.92, specificity 0.73-0.78), and calibration. Bootstrapping over ROC-AUC showed negligible biases (1.0 × 10-6 -1.23 × 10-3 ). In external validation, nFASS performed best with higher ROC-AUC. nFASS was strongly correlated (R 0.89) to best-worst scoring of 334 expert clinicians. CONCLUSION: FASS is a validated and reliable method to measure severity of food allergic reactions. The ordinal and numerical versions that map onto each other are suitable for use by different stakeholders in different settings.
- MeSH
- alergeny MeSH
- lidé MeSH
- plocha pod křivkou MeSH
- potravinová alergie * diagnóza MeSH
- potraviny MeSH
- ROC křivka MeSH
- Check Tag
- lidé MeSH
- Publikační typ
- časopisecké články MeSH
- práce podpořená grantem MeSH
Manual visual review, annotation and categorization of electroencephalography (EEG) is a time-consuming task that is often associated with human bias and requires trained electrophysiology experts with specific domain knowledge. This challenge is now compounded by development of measurement technologies and devices allowing large-scale heterogeneous, multi-channel recordings spanning multiple brain regions over days, weeks. Currently, supervised deep-learning techniques were shown to be an effective tool for analyzing big data sets, including EEG. However, the most significant caveat in training the supervised deep-learning models in a clinical research setting is the lack of adequate gold-standard annotations created by electrophysiology experts. Here, we propose a semi-supervised machine learning technique that utilizes deep-learning methods with a minimal amount of gold-standard labels. The method utilizes a temporal autoencoder for dimensionality reduction and a small number of the expert-provided gold-standard labels used for kernel density estimating (KDE) maps. We used data from electrophysiological intracranial EEG (iEEG) recordings acquired in two hospitals with different recording systems across 39 patients to validate the method. The method achieved iEEG classification (Pathologic vs. Normal vs. Artifacts) results with an area under the receiver operating characteristic (AUROC) scores of 0.862 ± 0.037, 0.879 ± 0.042, and area under the precision-recall curve (AUPRC) scores of 0.740 ± 0.740, 0.714 ± 0.042. This demonstrates that semi-supervised methods can provide acceptable results while requiring only 100 gold-standard data samples in each classification category. Subsequently, we deployed the technique to 12 novel patients in a pseudo-prospective framework for detecting Interictal epileptiform discharges (IEDs). We show that the proposed temporal autoencoder was able to generalize to novel patients while achieving AUROC of 0.877 ± 0.067 and AUPRC of 0.705 ± 0.154.
OBJECTIVES: To develop and validate the cut-offs in the Juvenile DermatoMyositis Activity Index (JDMAI) to distinguish the states of inactive disease (ID), low disease activity (LDA), moderate disease activity (MDA) and high disease activity (HDA) in children with juvenile dermatomyositis (JDM). METHODS: For cut-off definition, data from 139 patients included in a randomised clinical trial were used. Among the six versions of the JDMAI, JDMA1 (score range 0-40) and JDMAI2 (score range 0-39) were selected. Optimal cut-offs were determined against external criteria by calculating different percentiles of score distribution and through receiver operating characteristic curve analysis. External criteria included the modified Pediatric Rheumatology International Trials Organization (PRINTO) criteria for clinically ID in JDM (for ID) and PRINTO levels of improvement in the clinical trial (for LDA and HDA). MDA cut-offs were set at the score interval between LDA and HDA cut-offs. Cut-off validation was conducted by assessing construct and discriminative ability in two cohorts including a total of 488 JDM patients. RESULTS: The calculated JDMAI1 cut-offs were ≤2.4 for ID, ≤6.6 for LDA, 6.7-11 for MDA and >11 for HDA. The calculated JDMAI2 cut-offs were ≤5.2 for ID, ≤8.5 for LDA, 8.6-11.3 for MDA and >11.3 for HDA. The cut-offs discriminated strongly among disease activity states defined subjectively by caring physicians and parents, parents' satisfaction or non-satisfaction with illness outcome, levels of pain, fatigue, physical functional impairment and physical well-being. CONCLUSIONS: Both JDMAI1 and JDMAI2 cut-offs revealed good metrologic properties in validation analyses and are, therefore, suited for application in clinical practice and research.
- MeSH
- dermatomyozitida * diagnóza MeSH
- dítě MeSH
- lékaři * MeSH
- lidé MeSH
- randomizované kontrolované studie jako téma MeSH
- revmatologie * MeSH
- ROC křivka MeSH
- stupeň závažnosti nemoci MeSH
- Check Tag
- dítě MeSH
- lidé MeSH
- Publikační typ
- časopisecké články MeSH
BACKGROUND: Component-resolved diagnosis (CRD) has revealed significant associations between IgE against individual allergens and severity of hazelnut allergy. Less attention has been given to combining them with clinical factors in predicting severity. AIM: To analyze associations between severity and sensitization patterns, patient characteristics and clinical history, and to develop models to improve predictive accuracy. METHODS: Patients reporting hazelnut allergy (n = 423) from 12 European cities were tested for IgE against individual hazelnut allergens. Symptoms (reported and during Double-blind placebo-controlled food challenge [DBPCFC]) were categorized in mild, moderate, and severe. Multiple regression models to predict severity were generated from clinical factors and sensitization patterns (CRD- and extract-based). Odds ratios (ORs) and areas under receiver-operating characteristic (ROC) curves (AUCs) were used to evaluate their predictive value. RESULTS: Cor a 9 and 14 were positively (OR 10.5 and 10.1, respectively), and Cor a 1 negatively (OR 0.14) associated with severe symptoms during DBPCFC, with AUCs of 0.70-073. Combining Cor a 1 and 9 improved this to 0.76. A model using a combination of atopic dermatitis (risk), pollen allergy (protection), IgE against Cor a 14 (risk) and walnut (risk) increased the AUC to 0.91. At 92% sensitivity, the specificity was 76.3%, and the positive and negative predictive values 62.2% and 95.7%, respectively. For reported symptoms, associations and generated models proved to be almost identical but weaker. CONCLUSION: A model combining CRD with clinical background and extract-based serology is superior to CRD alone in assessing the risk of severe reactions to hazelnut, particular in ruling out severe reactions.
- MeSH
- alergeny imunologie MeSH
- alergie na ořechy diagnóza imunologie MeSH
- antigeny rostlinné imunologie MeSH
- dvojitá slepá metoda MeSH
- imunoglobulin E krev MeSH
- lidé MeSH
- líska imunologie MeSH
- multivariační analýza MeSH
- plocha pod křivkou MeSH
- ROC křivka MeSH
- senzitivita a specificita MeSH
- Check Tag
- lidé MeSH
- Publikační typ
- časopisecké články MeSH
- práce podpořená grantem MeSH
- randomizované kontrolované studie MeSH
OBJECTIVE: We aimed to evaluate the validity of a MARSIPAN-guidance-adapted Early Warning System (MARSI MEWS) and compare it to the National Early Warning Score (NEWS) and an adapted version of the Physical Risk in Eating Disorders Index (PREDIX), to ascertain whether current practice is comparable to best-practice standards. METHODS: We collated 3,937 observations from 36 inpatients from Addenbrookes Hospital over 2017-2018 and used three independent raters to create a "gold standard" of deteriorating cases. We ascertained performance metrics (Receiver Operating Characteristic Area Under the curve) for MARSI MEWS, NEWS and PREDIX; we also tested the proof of concept of a machine-learning-based early-warning-system (ML-EWS) using cross-validation and out-of-sample prediction of cases. RESULTS: The MARSI MEWS system showed higher ROC AUC (0.916) compared to NEWS (0.828) or PREDIX (0.865). ML-EWS (random forest) performed well at independent samples analysis (0.980) and multilevel analysis (0.922). CONCLUSION: MARSI MEWS seems most suitable for identifying critically deteriorating cases in anorexia nervosa inpatient population. We did not examine community practice in which the PREDIX arguably remains the best to ascertain deteriorating cases. Our results also provide a first proof of concept for the development of artificial-intelligence-based early warning systems in anorexia nervosa. Implications for inpatient clinical practice in eating disorders are discussed.
- MeSH
- časná diagnóza * MeSH
- dospělí MeSH
- hospitalizace * MeSH
- klinické zhoršení * MeSH
- lidé MeSH
- mentální anorexie terapie MeSH
- monitorování fyziologických funkcí metody MeSH
- plocha pod křivkou MeSH
- reprodukovatelnost výsledků MeSH
- ROC křivka MeSH
- systém včasného varování MeSH
- Check Tag
- dospělí MeSH
- lidé MeSH
- mužské pohlaví MeSH
- ženské pohlaví MeSH
- Publikační typ
- časopisecké články MeSH
- pozorovací studie MeSH
- práce podpořená grantem MeSH
OBJECTIVE: The aim of the study was to investigate the associations of cross-sectional area (CSA) of the median nerve measured by ultrasonography, the median to ulnar nerve ratio (MUR), the median to ulnar nerve difference (MUD) and the ratio of CSA of the median nerve to height squared (MHS) in relation to electrodiagnostic classification of moderate and severe carpal tunnel syndrome (CTS) and thus to identify patients suitable for surgical treatment. MATERIALS AND METHODS: A prospective study was conducted in patients aged ≥ 18 years who underwent both median and ulnar nerve ultrasonography and electrodiagnostic studies (EDS). 124 wrists of 62 patients were examined. The patients' characteristics were acquired through a questionnaire. CTS was diagnosed using EDS and classified according to the guidelines of the Czech Republic Association of Electrodiagnostic Medicine. The CSA of the median nerve and of the ulnar nerve were measured at the carpal tunnel inlet. RESULTS: Median nerve CSA at the tunnel inlet ≥ 12 mm2 correlates with electrodiagnostic classification of moderate to severe carpal tunnel syndrome. At this cut-off value, the sensitivity of ultrasonography is 82.4%, its specificity is 87.7%, the positive predictive value is 82.4%, the negative predictive value is 87.7%. MUD, MUR and MHS perform worse than the median nerve CSA, as shown by their lower area under the receiver operating characteristic curve. CONCLUSIONS: Ultrasound could help us indicate surgical treatment for CTS, especially in patients with clinical findings. Our results suggest a cut-off value of CSA at the tunnel inlet of ≥ 12mm2.
The electroencephalogram (EEG) is a cornerstone of neurophysiological research and clinical neurology. Historically, the classification of EEG as showing normal physiological or abnormal pathological activity has been performed by expert visual review. The potential value of unbiased, automated EEG classification has long been recognized, and in recent years the application of machine learning methods has received significant attention. A variety of solutions using convolutional neural networks (CNN) for EEG classification have emerged with impressive results. However, interpretation of CNN results and their connection with underlying basic electrophysiology has been unclear. This paper proposes a CNN architecture, which enables interpretation of intracranial EEG (iEEG) transients driving classification of brain activity as normal, pathological or artifactual. The goal is accomplished using CNN with long short-term memory (LSTM). We show that the method allows the visualization of iEEG graphoelements with the highest contribution to the final classification result using a classification heatmap and thus enables review of the raw iEEG data and interpret the decision of the model by electrophysiology means.
- MeSH
- artefakty MeSH
- datové soubory jako téma MeSH
- deep learning * MeSH
- elektroencefalografie klasifikace přístrojové vybavení metody MeSH
- lidé MeSH
- ROC křivka MeSH
- Check Tag
- lidé MeSH
- Publikační typ
- časopisecké články MeSH
- pozorovací studie MeSH
- práce podpořená grantem MeSH
- Research Support, N.I.H., Extramural MeSH
- validační studie MeSH
There is no general consensus regarding which accelerometer cut-off point (CoP) is most acceptable to estimate the time spent in moderate-to-vigorous physical activity (MVPA) in children and choice of an appropriate CoP primarily remains a subjective decision. Therefore, this study aimed to analyze the influence of CoP selection on the mean MVPA and to define the optimal thresholds of MVPA derived from different accelerometer CoPs to avoid overweight/obesity and adiposity in children aged 7 to 12 years. Three hundred six children participated. Physical activity (PA) was monitored for seven consecutive days using an ActiGraph accelerometer (model GT3X) and the intensity of PA was estimated using the five most frequently published CoPs. Body adiposity was assessed using a multi-frequency bioelectrical impedance analysis. There was found a wide range of mean levels of MVPA that ranged from 27 (Puyau CoP) to 231 min∙d-1 (Freedson 2005 CoP). A receiver operating characteristic curve analysis indicated that the optimal thresholds for counts per minute (cpm) and MVPA derived from the Puyau CoP was the most useful in classifying children according to their body mass index (BMI) and fat mass percentage (FM%). In the total sample, the optimal thresholds of the MVPA derived from the Puyau CoP were 22 and 23 min∙d-1 when the categories based on BMI and FM%, respectively, were used. The children who did not meet these optimal thresholds had a significantly increased risk of being overweight/obese (OR = 2.88, P < 0.01) and risk of having excess fat mass (OR = 2.41, P < 0.01). In conclusion, the decision of selecting among various CoPs significantly influences the optimal levels of MVPA. The Puyau CoP of 3 200 cmp seems to be the most useful for defining the optimal level of PA for pediatric obesity prevention.
- MeSH
- adipozita MeSH
- akcelerometrie MeSH
- cvičení * MeSH
- dítě MeSH
- index tělesné hmotnosti MeSH
- lidé MeSH
- obezita dětí a dospívajících prevence a kontrola MeSH
- odds ratio MeSH
- plocha pod křivkou MeSH
- ROC křivka MeSH
- Check Tag
- dítě MeSH
- lidé MeSH
- mužské pohlaví MeSH
- ženské pohlaví MeSH
- Publikační typ
- časopisecké články MeSH
