BACKGROUND: Disability progression is a key milestone in the disease evolution of people with multiple sclerosis (PwMS). Prediction models of the probability of disability progression have not yet reached the level of trust needed to be adopted in the clinic. A common benchmark to assess model development in multiple sclerosis is also currently lacking. METHODS: Data of adult PwMS with a follow-up of at least three years from 146 MS centers, spread over 40 countries and collected by the MSBase consortium was used. With basic inclusion criteria for quality requirements, it represents a total of 15, 240 PwMS. External validation was performed and repeated five times to assess the significance of the results. Transparent Reporting for Individual Prognosis Or Diagnosis (TRIPOD) guidelines were followed. Confirmed disability progression after two years was predicted, with a confirmation window of six months. Only routinely collected variables were used such as the expanded disability status scale, treatment, relapse information, and MS course. To learn the probability of disability progression, state-of-the-art machine learning models were investigated. The discrimination performance of the models is evaluated with the area under the receiver operator curve (ROC-AUC) and under the precision recall curve (AUC-PR), and their calibration via the Brier score and the expected calibration error. All our preprocessing and model code are available at https://gitlab.com/edebrouwer/ms_benchmark, making this task an ideal benchmark for predicting disability progression in MS. FINDINGS: Machine learning models achieved a ROC-AUC of 0⋅71 ± 0⋅01, an AUC-PR of 0⋅26 ± 0⋅02, a Brier score of 0⋅1 ± 0⋅01 and an expected calibration error of 0⋅07 ± 0⋅04. The history of disability progression was identified as being more predictive for future disability progression than the treatment or relapses history. CONCLUSIONS: Good discrimination and calibration performance on an external validation set is achieved, using only routinely collected variables. This suggests machine-learning models can reliably inform clinicians about the future occurrence of progression and are mature for a clinical impact study.

1 Biostat Hasselt University Belgium

Academic MS Center Zuyderland Department of Neurology Zuyderland Medical Center Sittard Geleen The Netherlands

AHEPA University Hospital Thessaloniki Greece

American University of Beirut Medical Center Beirut Lebanon

Amiri Hospital Sharq Kuwait

AZ Alma Ziekenhuis Sijsele Damme Belgium

Azienda Ospedaliera di Rilievo Nazionale San Giuseppe Moscati Avellino Avellino Italy

Azienda Sanitaria Unica Regionale Marche AV3 Macerata Italy

Bakirkoy Education and Research Hospital for Psychiatric and Neurological Diseases Istanbul Turkey

BAZ County Hospital Miskolc Hungary

Biobix Department of Data Analysis and Mathematical Modelling Ghent University Belgium

Biomedical Research Institute Hasselt University Belgium

Box Hill Hospital Melbourne Australia

Brain Ghent University Belgium

Centro Hospitalar Universitario de Sao Joao Porto Portugal

Charles University Prague and General University Hospital Prague Czech Republic

CHUM and Université de Montreal Montreal Canada

CISSS Chaudière Appalache Levis Canada

Cliniques Universitaires Saint Luc Brussels Belgium

College of Medicine and Health Sciences and Sultan Qaboos University Hospital SQU Oman

Concord Repatriation General Hospital Sydney Australia

CORe Department of Medicine University of Melbourne Melbourne Australia

Data Science Institute Hasselt University Belgium

Department of Medical and Surgical Sciences and Advanced Technologies GF Ingrassia Catania Italy

Department of Neurology Buffalo General Medical Center Buffalo United States of America

Department of Neurology Ghent University Belgium

Dept of Rehabilitation CRFF Mons Luigi Novarese Moncrivello Italy

Emergency Clinical County Hospital Pius Brinzeu Timisoara Romania and University of Medicine and Pharmacy Victor Babes Timisoara Romania

ESAT STADIUS KU Leuven Belgium

Geneva University Hospital Geneva Switzerland

Groene Hart Ziekenhuis Gouda Netherlands

Hospital Clinic de Barcelona Barcelona Spain

Hospital de Galdakao Usansolo Galdakao Spain

Hospital Fernandez Capital Federal Argentina

Hospital General Universitario de Alicante Alicante Spain

Hospital Universitario Donostia San Sebastián Spain

Hospital Universitario Virgen Macarena Sevilla Spain

IRCCS Istituto delle Scienze Neurologiche di Bologna Bologna Italia and Dipartimento di Scienze Biomediche e Neuromotorie Università di Bologna Bologna Italia

Jahn Ferenc Teaching Hospital Budapest Hungary

KERMIT Department of Data Analysis and Mathematical Modelling Ghent University Belgium

King Fahad Specialist Hospital Dammam Khobar Saudi Arabia

Koc University School of Medicine Istanbul Turkey

Liverpool Hospital Sydney Australia

Mater Dei Hospital Msida Malta

Mayis University Samsun Turkey

Melbourne MS Centre Department of Neurology Royal Melbourne Hospital Melbourne Australia

MS center UOC Neurologia ARNAS Garibaldi Catania Italy

Nemocnice Jihlava Jihlava Czech Republic

Neuro Rive Sud Quebec Canada

Noorderhart ziekenhuizen Pelt Belgium

Ospedali Riuniti di Salerno Salerno Italy

Razi Hospital Manouba Tunisia

Royal Hobart Hospital Hobart Australia

Royal Victoria Hospital Belfast United Kingdom

School for Mental Health and Neuroscience Maastricht University Maastricht The Netherlands

Semmelweis University Budapest Budapest Hungary

South Eastern HSC Trust Belfast United Kingdom

St Michael's Hospital Toronto Canada

St Vincent's University Hospital Dublin Ireland

SUMO IDLAB Ghent University imec Belgium

The Alfred Hospital Melbourne Australia

Universidade Metropolitana de Santos Santos Brazil

Universitair MS Centrum Hasselt Pelt Belgium

Universitary Hospital Ghent Ghent Belgium

University Hospital Reina Sofia Cordoba Spain

University Newcastle Newcastle Australia

University of Debrecen Debrecen Hungary

University of Western Australia Nedlands Australia

See more in PubMed

Reich DS, Lucchinetti CF, Calabresi PA. Multiple Sclerosis. New England Journal of Medicine. 2018;378(2):169–180. doi: 10.1056/NEJMra1401483 PubMed DOI PMC

Walton C, King R, Rechtman L, Kaye W, Leray E, Marrie RA, et al.. Rising prevalence of multiple sclerosis worldwide: Insights from the Atlas of MS. Multiple Sclerosis Journal. 2020;26(14):1816–1821. PubMed PMC

Degenhardt A, Ramagopalan SV, Scalfari A, Ebers GC. Clinical prognostic factors in multiple sclerosis: a natural history review. Nature Reviews Neurology. 2009;5(12):672–682. doi: 10.1038/nrneurol.2009.178 PubMed DOI

Dennison L, Brown M, Kirby S, Galea I. Do people with multiple sclerosis want to know their prognosis? A UK nationwide study. PLOS ONE. 2018;13(2):1–14. doi: 10.1371/journal.pone.0193407 PubMed DOI PMC

Brown FS, Glasmacher SA, Kearns PKA, MacDougall N, Hunt D, Connick P, et al.. Systematic review of prediction models in relapsing remitting multiple sclerosis. PLOS ONE. 2020;15(5):1–13. doi: 10.1371/journal.pone.0233575 PubMed DOI PMC

Havas J, Leray E, Rollot F, Casey R, Michel L, Lejeune F, et al.. Predictive medicine in multiple sclerosis: A systematic review. Multiple Sclerosis and Related Disorders. 2020;40:101928. doi: 10.1016/j.msard.2020.101928 PubMed DOI

Seccia R, Romano S, Salvetti M, Crisanti A, Palagi L, Grassi F. Machine Learning Use for Prognostic Purposes in Multiple Sclerosis. Life. 2021;11(2). doi: 10.3390/life11020122 PubMed DOI PMC

Hartmann M, Fenton N, Dobson R. Current review and next steps for artificial intelligence in multiple sclerosis risk research. Computers in Biology and Medicine. 2021;132:104337. doi: 10.1016/j.compbiomed.2021.104337 PubMed DOI

Peeters LM, Parciak T, Kalra D, Moreau Y, Kasilingam E, Van Galen P, et al.. Multiple Sclerosis Data Alliance–A global multi-stakeholder collaboration to scale-up real world data research. Multiple Sclerosis and Related Disorders. 2021;47:102634. doi: 10.1016/j.msard.2020.102634 PubMed DOI

Tintore M, Rovira À, Río J, Otero-Romero S, Arrambide G, Tur C, et al.. Defining high, medium and low impact prognostic factors for developing multiple sclerosis. Brain. 2015;138(7):1863–1874. doi: 10.1093/brain/awv105 PubMed DOI

Kelly CJ, Karthikesalingam A, Suleyman M, Corrado G, King D. Key challenges for delivering clinical impact with artificial intelligence. BMC medicine. 2019;17(1):1–9. doi: 10.1186/s12916-019-1426-2 PubMed DOI PMC

Mateen BA, Liley J, Denniston AK, Holmes CC, Vollmer SJ. Improving the quality of machine learning in health applications and clinical research. Nature Machine Intelligence. 2020;2(10):554–556. doi: 10.1038/s42256-020-00239-1 DOI

Kalincik T, Manouchehrinia A, Sobisek L, Jokubaitis V, Spelman T, Horakova D, et al.. Towards personalized therapy for multiple sclerosis: prediction of individual treatment response. Brain. 2017;140(9):2426–2443. doi: 10.1093/brain/awx185 PubMed DOI

Chalkou K, Steyerberg E, Bossuyt P, Subramaniam S, Benkert P, Kuhle J, et al.. Development, validation and clinical usefulness of a prognostic model for relapse in relapsing-remitting multiple sclerosis. Diagnostic and prognostic research. 2021;5(1):1–16. doi: 10.1186/s41512-021-00106-6 PubMed DOI PMC

Butzkueven H, Chapman J, Cristiano E, Grand’Maison F, Hoffmann M, Izquierdo G, et al.. MSBase: an international, online registry and platform for collaborative outcomes research in multiple sclerosis. Multiple Sclerosis Journal. 2006;12(6):769–774. doi: 10.1177/1352458506070775 PubMed DOI

Kalincik T, Butzkueven H. The MSBase registry: Informing clinical practice. Multiple Sclerosis Journal. 2019;25(14):1828–1834. doi: 10.1177/1352458519848965 PubMed DOI

De Brouwer E, Becker T, Moreau Y, Havrdova EK, Trojano M, Eichau S, et al.. Longitudinal machine learning modeling of MS patient trajectories improves predictions of disability progression. Computer Methods and Programs in Biomedicine. 2021; p. 106180. 10.1016/j.cmpb.2021.106180 PubMed DOI

Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. Journal of Clinical Epidemiology. 2016;74:167–176. doi: 10.1016/j.jclinepi.2015.12.005 PubMed DOI

Altmann A, Toloşi L, Sander O, Lengauer T. Permutation importance: a corrected feature importance measure. Bioinformatics. 2010;26(10):1340–1347. doi: 10.1093/bioinformatics/btq134 PubMed DOI

Schlaeger R, D’Souza M, Schindler C, Grize L, Dellas S, Radue E, et al.. Prediction of long-term disability in multiple sclerosis. Multiple Sclerosis Journal. 2012;18(1):31–38. doi: 10.1177/1352458511416836 PubMed DOI

de Groot V, Beckerman H, Uitdehaag BM, Hintzen RQ, Minneboo A, Heymans MW, et al.. Physical and Cognitive Functioning After 3 Years Can Be Predicted Using Information From the Diagnostic Process in Recently Diagnosed Multiple Sclerosis. Archives of Physical Medicine and Rehabilitation. 2009;90(9):1478–1488. doi: 10.1016/j.apmr.2009.03.018 PubMed DOI

von Gumberz J, Mahmoudi M, Young K, Schippling S, Martin R, Heesen C, et al.. Short-term MRI measurements as predictors of EDSS progression in relapsing-remitting multiple sclerosis: grey matter atrophy but not lesions are predictive in a real-life setting. PeerJ. 2016;4:e2442. doi: 10.7717/peerj.2442 PubMed DOI PMC

Agniel D, Kohane IS, Weber GM. Biases in electronic health record data due to processes within the healthcare system: retrospective observational study. BMJ. 2018;361. doi: 10.1136/bmj.k1479 PubMed DOI PMC

Goodkin DE, Cookfair D, Wende K, Bourdette D, Pullicino P, Scherokman B, et al.. Inter- and intrarater scoring agreement using grades 1.0 to 3.5 of the Kurtzke Expanded Disability Status Scale (EDSS). Neurology. 1992;42(4):859–859. doi: 10.1212/WNL.42.4.859 PubMed DOI

Kalincik T, Cutter G, Spelman T, Jokubaitis V, Havrdova E, Horakova D, et al.. Defining reliable disability outcomes in multiple sclerosis. Brain. 2015;138(11):3287–3298. doi: 10.1093/brain/awv258 PubMed DOI

Robertson D, Moreo N. Disease-modifying therapies in multiple sclerosis: overview and treatment considerations. Federal Practitioner. 2016;33(6):28. PubMed PMC

Esteva A, Robicquet A, Ramsundar B, Kuleshov V, DePristo M, Chou K, et al.. A guide to deep learning in healthcare. Nature medicine. 2019;25(1):24–29. doi: 10.1038/s41591-018-0316-z PubMed DOI

De Brouwer E, Gonzalez J, Hyland S. Predicting the impact of treatments over time with uncertainty aware neural differential equations. In: International Conference on Artificial Intelligence and Statistics. PMLR; 2022. p. 4705–4722.

De Brouwer E, Krishnan RG. Anamnesic Neural Differential Equations with Orthogonal Polynomial Projections. In: The Eleventh International Conference on Learning Representations; 2022.

Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need. In: Advances in neural information processing systems; 2017. p. 5998–6008.

Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) the TRIPOD statement. Circulation. 2015;131(2):211–219. doi: 10.1161/CIRCULATIONAHA.114.014508 PubMed DOI PMC

Gal Y, Ghahramani Z. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In: international conference on machine learning. PMLR; 2016. p. 1050–1059.

Platt JC. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers. 1999;10(3):61–74.

Zadrozny B, Elkan C. Transforming classifier scores into accurate multiclass probability estimates. In: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining; 2002. p. 694–699.