Inter-rater reliability (IRR) is one of the commonly used tools for assessing the quality of ratings from multiple raters. However, applicant selection procedures based on ratings from multiple raters usually result in a binary outcome; the applicant is either selected or not. This final outcome is not considered in IRR, which instead focuses on the ratings of the individual subjects or objects. We outline the connection between the ratings' measurement model (used for IRR) and a binary classification framework. We develop a simple way of approximating the probability of correctly selecting the best applicants which allows us to compute error probabilities of the selection procedure (i.e., false positive and false negative rate) or their lower bounds. We draw connections between the IRR and the binary classification metrics, showing that binary classification metrics depend solely on the IRR coefficient and proportion of selected applicants. We assess the performance of the approximation in a simulation study and apply it in an example comparing the reliability of multiple grant peer review selection procedures. We also discuss other possible uses of the explored connections in other contexts, such as educational testing, psychological assessment, and health-related measurement, and implement the computations in the R package IRR2FPR.
- Klíčová slova
- Type I error, Type II error, error rate, mixed‐effect models, rating,
- MeSH
- falešně pozitivní reakce MeSH
- lidé MeSH
- odchylka pozorovatele MeSH
- počítačová simulace MeSH
- posudkové řízení metody MeSH
- pravděpodobnost MeSH
- reprodukovatelnost výsledků MeSH
- statistické modely * MeSH
- Check Tag
- lidé MeSH
- Publikační typ
- časopisecké články MeSH
The influential claim that most published results are false raised concerns about the trustworthiness and integrity of science. Since then, there have been numerous attempts to examine the rate of false-positive results that have failed to settle this question empirically. Here we propose a new way to estimate the false positive risk and apply the method to the results of (randomized) clinical trials in top medical journals. Contrary to claims that most published results are false, we find that the traditional significance criterion of α = .05 produces a false positive risk of 13%. Adjusting α to.01 lowers the false positive risk to less than 5%. However, our method does provide clear evidence of publication bias that leads to inflated effect size estimates. These results provide a solid empirical foundation for evaluations of the trustworthiness of medical research.
- MeSH
- biomedicínský výzkum * MeSH
- insuflace * MeSH
- periodika jako téma * MeSH
- Publikační typ
- časopisecké články MeSH
Publication bias is a ubiquitous threat to the validity of meta-analysis and the accumulation of scientific evidence. In order to estimate and counteract the impact of publication bias, multiple methods have been developed; however, recent simulation studies have shown the methods' performance to depend on the true data generating process, and no method consistently outperforms the others across a wide range of conditions. Unfortunately, when different methods lead to contradicting conclusions, researchers can choose those methods that lead to a desired outcome. To avoid the condition-dependent, all-or-none choice between competing methods and conflicting results, we extend robust Bayesian meta-analysis and model-average across two prominent approaches of adjusting for publication bias: (1) selection models of p-values and (2) models adjusting for small-study effects. The resulting model ensemble weights the estimates and the evidence for the absence/presence of the effect from the competing approaches with the support they receive from the data. Applications, simulations, and comparisons to preregistered, multi-lab replications demonstrate the benefits of Bayesian model-averaging of complementary publication bias adjustment methods.
The SARS-CoV-2 pandemic is not only a threat to physical health but is also having severe impacts on mental health. Although increases in stress-related symptomatology and other adverse psycho-social outcomes, as well as their most important risk factors have been described, hardly anything is known about potential protective factors. Resilience refers to the maintenance of mental health despite adversity. To gain mechanistic insights about the relationship between described psycho-social resilience factors and resilience specifically in the current crisis, we assessed resilience factors, exposure to Corona crisis-specific and general stressors, as well as internalizing symptoms in a cross-sectional online survey conducted in 24 languages during the most intense phase of the lockdown in Europe (22 March to 19 April) in a convenience sample of N = 15,970 adults. Resilience, as an outcome, was conceptualized as good mental health despite stressor exposure and measured as the inverse residual between actual and predicted symptom total score. Preregistered hypotheses (osf.io/r6btn) were tested with multiple regression models and mediation analyses. Results confirmed our primary hypothesis that positive appraisal style (PAS) is positively associated with resilience (p < 0.0001). The resilience factor PAS also partly mediated the positive association between perceived social support and resilience, and its association with resilience was in turn partly mediated by the ability to easily recover from stress (both p < 0.0001). In comparison with other resilience factors, good stress response recovery and positive appraisal specifically of the consequences of the Corona crisis were the strongest factors. Preregistered exploratory subgroup analyses (osf.io/thka9) showed that all tested resilience factors generalize across major socio-demographic categories. This research identifies modifiable protective factors that can be targeted by public mental health efforts in this and in future pandemics.
- MeSH
- COVID-19 prevence a kontrola psychologie MeSH
- dospělí MeSH
- duševní zdraví * MeSH
- lidé středního věku MeSH
- lidé MeSH
- mladý dospělý MeSH
- multivariační analýza MeSH
- ochranné faktory MeSH
- přenos infekční nemoci prevence a kontrola MeSH
- průřezové studie MeSH
- psychická odolnost * MeSH
- psychický stres prevence a kontrola MeSH
- regresní analýza MeSH
- sociální faktory * MeSH
- sociální opora MeSH
- Check Tag
- dospělí MeSH
- lidé středního věku MeSH
- lidé MeSH
- mladý dospělý MeSH
- mužské pohlaví MeSH
- ženské pohlaví MeSH
- Publikační typ
- časopisecké články MeSH
- práce podpořená grantem MeSH
- Geografické názvy
- Evropa MeSH
BACKGROUND: Childhood adversity (CA) is strongly associated with mental health problems. Resilience factors (RFs) reduce mental health problems following CA. Yet, knowledge on the nature of RFs is scarce. Therefore, we examined RF mean levels, RF interrelations, RF-distress pathways, and their changes between early (age 14) and later adolescence (age 17). METHODS: We studied 10 empirically supported RFs in adolescents with (CA+; n = 631) and without CA (CA-; n = 499), using network psychometrics. RESULTS: All inter-personal RFs (e.g. friendships) showed stable mean levels between age 14 and 17, and three of seven intra-personal RFs (e.g. distress tolerance) changed in a similar manner in the two groups. The CA+ group had lower RFs and higher distress at both ages. Thus, CA does not seem to inhibit RF changes, but to increase the risk of persistently lower RFs. At age 14, but not 17, the RF network of the CA+ group was less positively connected, suggesting that RFs are less likely to enhance each other than in the CA- group. Those findings underpin the notion that CA has a predominantly strong proximal effect. RF-distress pathways did not differ in strength between the CA+ and the CA- group, which suggests that RFs have a similarly protective strength in the two groups. Yet, as RFs are lower and distress is higher, RF-distress pathways may overall be less advantageous in the CA+ group. Most RF interrelations and RF-distress pathways were stable between age 14 and 17, which may help explain why exposure to CA is frequently found to have a lasting impact on mental health. CONCLUSIONS: Our findings not only shed light on the nature and changes of RFs between early and later adolescence, but also offer some accounts for why exposure to CA has stronger proximal effects and is often found to have a lasting impact on mental health.
- Klíčová slova
- Adolescence, Childhood adversity, Mental health, Resilience factors,
- MeSH
- fyziologický stres MeSH
- lidé MeSH
- mladiství MeSH
- psychická odolnost * MeSH
- psychický stres MeSH
- vývoj mladistvých * MeSH
- Check Tag
- lidé MeSH
- mladiství MeSH
- mužské pohlaví MeSH
- ženské pohlaví MeSH
- Publikační typ
- časopisecké články MeSH
- práce podpořená grantem MeSH
