OBJECTIVE: To externally validate various prognostic models and scoring rules for predicting short term mortality in patients admitted to hospital for covid-19. DESIGN: Two stage individual participant data meta-analysis. SETTING: Secondary and tertiary care. PARTICIPANTS: 46 914 patients across 18 countries, admitted to a hospital with polymerase chain reaction confirmed covid-19 from November 2019 to April 2021. DATA SOURCES: Multiple (clustered) cohorts in Brazil, Belgium, China, Czech Republic, Egypt, France, Iran, Israel, Italy, Mexico, Netherlands, Portugal, Russia, Saudi Arabia, Spain, Sweden, United Kingdom, and United States previously identified by a living systematic review of covid-19 prediction models published in The BMJ, and through PROSPERO, reference checking, and expert knowledge. MODEL SELECTION AND ELIGIBILITY CRITERIA: Prognostic models identified by the living systematic review and through contacting experts. A priori models were excluded that had a high risk of bias in the participant domain of PROBAST (prediction model study risk of bias assessment tool) or for which the applicability was deemed poor. METHODS: Eight prognostic models with diverse predictors were identified and validated. A two stage individual participant data meta-analysis was performed of the estimated model concordance (C) statistic, calibration slope, calibration-in-the-large, and observed to expected ratio (O:E) across the included clusters. MAIN OUTCOME MEASURES: 30 day mortality or in-hospital mortality. RESULTS: Datasets included 27 clusters from 18 different countries and contained data on 46 914patients. The pooled estimates ranged from 0.67 to 0.80 (C statistic), 0.22 to 1.22 (calibration slope), and 0.18 to 2.59 (O:E ratio) and were prone to substantial between study heterogeneity. The 4C Mortality Score by Knight et al (pooled C statistic 0.80, 95% confidence interval 0.75 to 0.84, 95% prediction interval 0.72 to 0.86) and clinical model by Wang et al (0.77, 0.73 to 0.80, 0.63 to 0.87) had the highest discriminative ability. On average, 29% fewer deaths were observed than predicted by the 4C Mortality Score (pooled O:E 0.71, 95% confidence interval 0.45 to 1.11, 95% prediction interval 0.21 to 2.39), 35% fewer than predicted by the Wang clinical model (0.65, 0.52 to 0.82, 0.23 to 1.89), and 4% fewer than predicted by Xie et al's model (0.96, 0.59 to 1.55, 0.21 to 4.28). CONCLUSION: The prognostic value of the included models varied greatly between the data sources. Although the Knight 4C Mortality Score and Wang clinical model appeared most promising, recalibration (intercept and slope updates) is needed before implementation in routine care.

Bernhoven Uden Netherlands

Centre for Access and Delivery Research Evaluation Iowa City Veterans Affairs Health Care System Iowa City IA USA

Centre for Clinical Infection and Diagnostics Research School of Immunology and Microbial Sciences King's College London London UK

Cochrane Netherlands University Medical Centre Utrecht Utrecht University Netherlands

Copenhagen Trial Unit Centre for Clinical Intervention Research Department 7812 Rigshospitalet Copenhagen University Hospital Denmark

Data Analytics and Methods Task Force European Medicines Agency Amsterdam Netherlands

Department of Biomedical Data Sciences Leiden University Medical Centre Leiden Netherlands

Department of Cardiology Division of Heart and Lungs University Medical Centre Utrecht Utrecht University Utrecht Netherlands

Department of Cardiovascular Sciences College of Life Sciences University of Leicester Leicester UK

Department of Development and Regeneration KU Leuven Leuven Belgium

Department of Epidemiology CAPHRI Care and Public Health Research Institute Maastricht University Maastricht Netherlands

Department of General Medicine Shirakawa Satellite for Teaching And Research Fukushima Medical University Fukushima Japan

Department of Health Technology and Services Research Technical Medical Centre University of Twente Enschede Netherlands

Department of Infectious Diseases Karolinska University Hospital Stockholm Sweden

Department of Pharmacy University Hospital Centre of Nîmes Nîmes France

Department of Respiratory Medicine University Hospitals of Leicester NHS Trust Leicester UK

Department of Respiratory Sciences College of Life Sciences University of Leicester Leicester UK

Dirección de Investigación Instituto Nacional de Geriatría Mexico City Mexico

Division of Infection and Immunity University College London London UK

Division of Infectious Diseases Department of Medicine Solna Karolinska Institute Stockholm Sweden

EPI centre KU Leuven Leuven Belgium

Faculty of Chemistry Universidad Nacional Autónoma de México México City Mexico

Health Data Research UK and Institute of Health Informatics University College London London UK

Heidelberger Institut für Global Health Universitätsklinikum Heidelberg Germany

Industrial Engineering Department Universidade Federal do Rio Grande do Sul Porto Alegre Brazil

Infectious Diseases Service UnityPoint Health Des Moines Des Moines IA USA

Institute for Global Health University College London London UK

Institute of Cardiovascular Science Faculty of Population Health Sciences University College London London UK

Institute of Microbiology of the Czech Academy of Sciences Prague Czech Republic

John Walls Renal Unit University Hospitals of Leicester NHS Trust Leicester UK

Julius Center for Health Sciences and Primary Care University Medical Centre Utrecht Utrecht University Utrecht Netherlands

Laboratory of Clinical Chemistry and Haematology Jeroen Bosch Hospital Den Bosch Netherlands

Maxima MC Veldhoven the Netherlands

MD PhD Program Faculty of Medicine National Autonomous University of Mexico Mexico City Mexico

NIHR Leicester Biomedical Research Centre University of Leicester Leicester UK

School of Population Health and Environmental Sciences King's College London London UK

Section of Epidemiology Department of Public Health University of Copenhagen Copenhagen Denmark

University of Iowa Carver College of Medicine Iowa City IA USA

See more in PubMed

Dong E, Du H, Gardner L. An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect Dis 2020;20:533-4. 10.1016/S1473-3099(20)30120-1 PubMed DOI PMC

Ong SWX, Young BE, Lye DC. Lack of detail in population-level data impedes analysis of SARS-CoV-2 variants of concern and clinical outcomes. Lancet Infect Dis 2021;21:1195-7. 10.1016/S1473-3099(21)00201-2 PubMed DOI PMC

Samadizadeh S, Masoudi M, Rastegar M, Salimi V, Shahbaz MB, Tahamtan A. COVID-19: Why does disease severity vary among individuals? Respir Med 2021;180:106356. https://www.resmedjournal.com/article/S0954-6111(21)00062-7/abstract. 10.1016/j.rmed.2021.106356 PubMed DOI PMC

Smith GD, Spiegelhalter D. Shielding from covid-19 should be stratified by risk. BMJ 2020;369:m2063. 10.1136/bmj.m2063 PubMed DOI

Wynants L, Van Calster B, Collins GS, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal (update 2, 21 July 2020) PubMed DOI PMC

Siontis GCM, Tzoulaki I, Castaldi PJ, Ioannidis JPA. External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol 2015;68:25-34. 10.1016/j.jclinepi.2014.09.007 PubMed DOI

Altman DG, Vergouwe Y, Royston P, Moons KGM. Prognosis and prognostic research: validating a prognostic model. BMJ 2009;338:b605. 10.1136/bmj.b605 PubMed DOI

Moons KGM, Kengne AP, Grobbee DE, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart 2012;98:691-8. 10.1136/heartjnl-2011-301247 PubMed DOI

Bleeker SE, Moll HA, Steyerberg EW, et al. External validation is necessary in prediction research: a clinical example. J Clin Epidemiol 2003;56:826-32. 10.1016/S0895-4356(03)00207-5 PubMed DOI

Riley RD, van der Windt D, Croft P, Moons KGM. Prognosis Research in Healthcare: Concepts, Methods, and Impact. Oxford University Press, 2019. 10.1093/med/9780198796619.001.0001. DOI

Steyerberg EW. Clinical Prediction Models: a Practical Approach to Development, Validation, and Updating. Springer Science & Business Media, 2009. 10.1007/978-0-387-77244-8. DOI

Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med 2000;19:453-73. 10.1002/(SICI)1097-0258(20000229)19:4<453::AID-SIM350>3.0.CO;2-5 PubMed DOI

Wynants L, Van Calster B, Collins GS, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal (update 3, 12 January 2021) PubMed DOI PMC

Bello-Chavolla OY, Bahena-López JP, Antonio-Villa NE, et al. Predicting Mortality Due to SARS-CoV-2: A Mechanistic Score Relating Obesity and Diabetes to COVID-19 Outcomes in Mexico. J Clin Endocrinol Metab 2020;105:2752-61. 10.1210/clinem/dgaa346 PubMed DOI PMC

Riley RD, Lambert PC, Abo-Zaid G. Meta-analysis of individual participant data: rationale, conduct, and reporting. BMJ 2010;340:c221. 10.1136/bmj.c221 PubMed DOI

Debray TPA, Moons KGM, Abo-Zaid GMA, Koffijberg H, Riley RD. Individual participant data meta-analysis for a binary outcome: one-stage or two-stage? PLoS One 2013;8:e60650. PubMed PMC

The CAPACITY-COVID Collaborative Consortium and LEOSS Study Group . Clinical presentation, disease course, and outcome of COVID-19 in hospitalized patients with and without pre-existing cardiac disease: a cohort study across 18 countries. Eur Heart J 2021;ehab656. PubMed

Debray TPA, Damen JAAG, Snell KIE, et al. A guide to systematic review and meta-analysis of prediction model performance. BMJ 2017;356:i6460. 10.1136/bmj.i6460 PubMed DOI

Debray TPA, Damen JAAG, Riley RD, et al. A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes. Stat Methods Med Res 2019;28:2768-86. 10.1177/0962280218785504 PubMed DOI PMC

Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143:29-36. 10.1148/radiology.143.1.7063747 PubMed DOI

Cox DR. Two Further Applications of a Model for Binary Regression. Biometrika 1958;45:562-5 10.1093/biomet/45.3-4.562. DOI

Snell KIE, Ensor J, Debray TPA, Moons KGM, Riley RD. Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C-statistic and calibration measures? Stat Methods Med Res 2018;27:3505-22. 10.1177/0962280217705678 PubMed DOI PMC

Knapp G, Hartung J. Improved tests for a random effects meta-regression with a single covariate. Stat Med 2003;22:2693-710. 10.1002/sim.1482 PubMed DOI

Sidik K, Jonkman JN. Robust variance estimation for random effects meta-analysis. Comput Stat Data Anal 2006;50:3681-701 10.1016/j.csda.2005.07.019. DOI

Higgins JPT, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. J R Stat Soc Ser A Stat Soc 2009;172:137-59. 10.1111/j.1467-985X.2008.00552.x PubMed DOI PMC

Buuren S, Groothuis-Oudshoorn K. mice: Multivariate imputation by chained equations in R. J Stat Softw 2011;45. https://doc.utwente.nl/78938/ 10.18637/jss.v045.i03. DOI

R Core Team. R: A Language and Environment for Statistical Computing. 2021. https://www.R-project.org/. Accessed November, 9, 2021.

StataCorp . Stata Statistical Software: Release 17. StataCorp, 2021.

Debray TPA, de Jong VMT. metamisc: Meta-Analysis of Diagnosis and Prognosis Research Studies. 2021. https://cran.r-project.org/package=metamisc

Robin X, Turck N, Hainard A, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 2011;12:77. 10.1186/1471-2105-12-77 PubMed DOI PMC

Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med 2015;13:1. 10.1186/s12916-014-0241-z PubMed DOI PMC

Moons KGM, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015;162:W1-73. 10.7326/M14-0698 PubMed DOI

Xie J, Hungerford D, Chen H, et al. Development and external validation of a prognostic multivariable model on admission for hospitalized patients with COVID-19. medRxiv 2020;2020.03.28.20045997.

Hu C, Liu Z, Jiang Y, et al. Early prediction of mortality risk among severe COVID-19 patients using machine learning. medRxiv 2020;2020.04.13.20064329 10.1101/2020.04.13.20064329. DOI

Zhang H, Shi T, Wu X, et al. Risk prediction for poor outcome and death in hospital in-patients with COVID-19: derivation in Wuhan, China and external validation in London, UK. medRxiv 2020;2020.04.28.20082222.

Knight SR, Ho A, Pius R, et al. ISARIC4C investigators . Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score. BMJ 2020;370:m3339. 10.1136/bmj.m3339 PubMed DOI PMC

Wang K, Zuo P, Liu Y, et al. Clinical and Laboratory Predictors of In-hospital Mortality in Patients With Coronavirus Disease-2019: A Cohort Study in Wuhan, China. Clin Infect Dis 2020;71:2079-88. 10.1093/cid/ciaa538 PubMed DOI PMC

Takada T, Hoogland J,, van Lieshout C, et al. Accuracy of approximations to recover incompletely reported logistic regression models depended on other available information. J Clin Epidemiol 2022;143:81-90. https://www.jclinepi.com/article/S0895-4356(21)00391-7/fulltext. PubMed

Yan L, Zhang HT, Goncalves J, et al. An interpretable mortality prediction model for COVID-19 patients. Nat Mach Intell 2020;2:283-8 10.1038/s42256-020-0180-7. DOI

Toll DB, Janssen KJM, Vergouwe Y, Moons KGM. Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol 2008;61:1085-94. 10.1016/j.jclinepi.2008.04.008 PubMed DOI

Su TL, Jaki T, Hickey GL, Buchan I, Sperrin M. A review of statistical updating methods for clinical prediction models. Stat Methods Med Res 2018;27:185-97. 10.1177/0962280215626466 PubMed DOI

Jenkins DA, Sperrin M, Martin GP, Peek N. Dynamic models to predict health outcomes: current status and methodological challenges. Diagn Progn Res 2018;2:23. 10.1186/s41512-018-0045-2 PubMed DOI PMC

Jenkins DA, Martin GP, Sperrin M, et al. Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems? Diagn Progn Res 2021;5:1. 10.1186/s41512-020-00090-3 PubMed DOI PMC

Nguyen TL, Collins GS, Landais P, Le Manach Y. Counterfactual clinical prediction models could help to infer individualized treatment effects in randomized controlled trials-An illustration with the International Stroke Trial. J Clin Epidemiol 2020;125:47-56. 10.1016/j.jclinepi.2020.05.022 PubMed DOI

Hoogland J, IntHout J, Belias M, Rovers MM, Riley RD, Harrell FE. Jret al. A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint. Stat Med 2021;40:5961-81. https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.9154 PubMed DOI PMC