BACKGROUND: Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here, we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022. METHODS: We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported by a standardised source for 32 countries over the next 1-4 weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models' predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models' forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models' past predictive performance. RESULTS: Over 52 weeks, we collected forecasts from 48 unique models. We evaluated 29 models' forecast scores in comparison to the ensemble model. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 83% of participating models' forecasts of incident cases (with a total N=886 predictions from 23 unique models), and 91% of participating models' forecasts of deaths (N=763 predictions from 20 models). Across a 1-4 week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over 4 weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models. CONCLUSIONS: Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than 2 weeks. FUNDING: AA, BH, BL, LWa, MMa, PP, SV funded by National Institutes of Health (NIH) Grant 1R01GM109718, NSF BIG DATA Grant IIS-1633028, NSF Grant No.: OAC-1916805, NSF Expeditions in Computing Grant CCF-1918656, CCF-1917819, NSF RAPID CNS-2028004, NSF RAPID OAC-2027541, US Centers for Disease Control and Prevention 75D30119C05935, a grant from Google, University of Virginia Strategic Investment Fund award number SIF160, Defense Threat Reduction Agency (DTRA) under Contract No. HDTRA1-19-D-0007, and respectively Virginia Dept of Health Grant VDH-21-501-0141, VDH-21-501-0143, VDH-21-501-0147, VDH-21-501-0145, VDH-21-501-0146, VDH-21-501-0142, VDH-21-501-0148. AF, AMa, GL funded by SMIGE - Modelli statistici inferenziali per governare l'epidemia, FISR 2020-Covid-19 I Fase, FISR2020IP-00156, Codice Progetto: PRJ-0695. AM, BK, FD, FR, JK, JN, JZ, KN, MG, MR, MS, RB funded by Ministry of Science and Higher Education of Poland with grant 28/WFSN/2021 to the University of Warsaw. BRe, CPe, JLAz funded by Ministerio de Sanidad/ISCIII. BT, PG funded by PERISCOPE European H2020 project, contract number 101016233. CP, DL, EA, MC, SA funded by European Commission - Directorate-General for Communications Networks, Content and Technology through the contract LC-01485746, and Ministerio de Ciencia, Innovacion y Universidades and FEDER, with the project PGC2018-095456-B-I00. DE., MGu funded by Spanish Ministry of Health / REACT-UE (FEDER). DO, GF, IMi, LC funded by Laboratory Directed Research and Development program of Los Alamos National Laboratory (LANL) under project number 20200700ER. DS, ELR, GG, NGR, NW, YW funded by National Institutes of General Medical Sciences (R35GM119582; the content is solely the responsibility of the authors and does not necessarily represent the official views of NIGMS or the National Institutes of Health). FB, FP funded by InPresa, Lombardy Region, Italy. HG, KS funded by European Centre for Disease Prevention and Control. IV funded by Agencia de Qualitat i Avaluacio Sanitaries de Catalunya (AQuAS) through contract 2021-021OE. JDe, SMo, VP funded by Netzwerk Universitatsmedizin (NUM) project egePan (01KX2021). JPB, SH, TH funded by Federal Ministry of Education and Research (BMBF; grant 05M18SIA). KH, MSc, YKh funded by Project SaxoCOV, funded by the German Free State of Saxony. Presentation of data, model results and simulations also funded by the NFDI4Health Task Force COVID-19 (https://www.nfdi4health.de/task-force-covid-19-2) within the framework of a DFG-project (LO-342/17-1). LP, VE funded by Mathematical and Statistical modelling project (MUNI/A/1615/2020), Online platform for real-time monitoring, analysis and management of epidemic situations (MUNI/11/02202001/2020); VE also supported by RECETOX research infrastructure (Ministry of Education, Youth and Sports of the Czech Republic: LM2018121), the CETOCOEN EXCELLENCE (CZ.02.1.01/0.0/0.0/17-043/0009632), RECETOX RI project (CZ.02.1.01/0.0/0.0/16-013/0001761). NIB funded by Health Protection Research Unit (grant code NIHR200908). SAb, SF funded by Wellcome Trust (210758/Z/18/Z).

3rd Faculty of Medicine Charles University Prague Czech Republic

Boston Children's Hospital and Harvard Medical School Boston United States

Ecole Polytechnique Federale de Lausanne Lausanne Switzerland

Éducation nationale Valbonne France

Eidgenossische Technische Hochschule Zurich Switzerland

European Centre for Disease Prevention and Control Stockholm Sweden

Forschungszentrum Jülich GmbH Jülich Germany

Frankfurt Institute for Advanced Studies Frankfurt Germany

Fraunhofer Institute for Industrial Mathematics Kaiserslautern Germany

Heidelberg University Heidelberg Germany

Helmholtz Centre for Infection Research Braunschweig Germany

IEM Inc Baton Rouge United States

IEM Inc Bel Air United States

Independent researcher Davis United States

Independent researcher Vienna Austria

Institut d'Investigacions Biomèdiques August Pi i Sunyer Universitat Pompeu Fabra Barcelona Spain

Institute of Computer Science of the CAS Prague Czech Republic

Institute of Information Theory and Automation of the CAS Prague Czech Republic

Inverence Madrid Spain

Karlsruhe Institute of Technology Karlsruhe Germany

London School of Hygiene and Tropical Medicine London United Kingdom

Los Alamos National Laboratory Los Alamos United States

LUMSA University Rome Italy

Masaryk University Brno Czech Republic

Massachusetts Institute of Technology Cambridge United States

Max Planck Institut für Dynamik und Selbstorganisation Göttingen Germany

Medical University of Gdansk Gdańsk Poland

Paul Scherrer Institute Villigen Switzerland

Politecnico di Milano Milan Italy

Robert Koch Institute Berlin Germany

Technical University of Kaiserlautern Kaiserslautern Germany

Technische Universität Ilmenau Ilmenau Germany

Universidad Carlos 3 de Madrid Leganes Spain

Universidad Nacional de Educación a Distancia Madrid Spain

Universitat de Barcelona Barcelona Spain

Universitat Politècnica de Catalunya Barcelona Spain

Universitat Trier Trier Germany

University of Bialystok Warsaw Poland

University of Cologne Cologne Germany

University of Halle Halle Germany

University of Ljubljana Ljubljana Slovenia

University of Massachusetts Amherst Amherst United States

University of Milano Bicocca Milano Italy

University of Molise Pesche Italy

University of Oxford Oxford United Kingdom

University of Palermo Palermo Italy

University of Pavia Pavia Italy

University of Perugia Perugia Italy

University of Rome La Sapienza Rome Italy

University of Rome Tor Vergata Rome Italy

University of Southern California Los Angeles United States

University of Sydney Sydney Australia

University of Virginia Charlottesville United States

University of Warsaw Warsaw Poland

University of Wroclaw Wroclaw Poland

Universtät Leipzig Leipzig Germany

Warsaw University of Technology Warsaw Poland

Wroclaw University of Science and Technology Wroclaw Poland

Před aktualizací

doi: 10.1101/2022.06.16.22276024 PubMed

Zobrazit více v PubMed

Adib K, Hancock PA, Rahimli A, Mugisa B, Abdulrazeq F, Aguas R, White LJ, Hajjeh R, Al Ariqi L, Nabeth P. A participatory modelling approach for investigating the spread of covid-19 in countries of the eastern Mediterranean region to support public health decision-making. BMJ Global Health. 2021;6:e005207. doi: 10.1136/bmjgh-2021-005207. PubMed DOI PMC

Agosto A, Giudici P. A poisson autoregressive model to understand COVID-19 contagion dynamics. Risks. 2020;8:77. doi: 10.3390/risks8030077. DOI

Agosto A, Campmas A, Giudici P, Renda A. Monitoring COVID-19 contagion growth. Statistics in Medicine. 2021;40:4150–4160. doi: 10.1002/sim.9020. PubMed DOI PMC

Aguas R, White L, Hupert N, Shretta R, Pan-Ngum W, Celhay O, Moldokmatova A, Arifi F, Mirzazadeh A, Sharifi H, Adib K, Sahak MN, Franco C, Coutinho R, CoMo Consortium Modelling the COVID-19 pandemic in context: an international participatory approach. BMJ Global Health. 2020;5:e003126. doi: 10.1136/bmjgh-2020-003126. PubMed DOI PMC

Alene M, Yismaw L, Assemie MA, Ketema DB, Gietaneh W, Birhan TY. Serial interval and incubation period of COVID-19: a systematic review and meta-analysis. BMC Infectious Diseases. 2021;21:257. doi: 10.1186/s12879-021-05950-x. PubMed DOI PMC

Bicher M, Zuba M, Rainer L, Bachner F, Rippinger C, Ostermann H, Popper N, Thurner S, Klimek P. Supporting COVID-19 Policy-Making with a Predictive Epidemiological Multi-Model Warning System. medRxiv. 2020 doi: 10.1101/2020.10.18.20214767. PubMed DOI PMC

Borchering RK, Viboud C, Howerton E, Smith CP, Truelove S, Runge MC, Reich NG, Contamin L, Levander J, Salerno J, van Panhuis W, Kinsey M, Tallaksen K, Obrecht RF, Asher L, Costello C, Kelbaugh M, Wilson S, Shin L, Gallagher ME, Mullany LC, Rainwater-Lovett K, Lemaitre JC, Dent J, Grantz KH, Kaminsky J, Lauer SA, Lee EC, Meredith HR, Perez-Saez J, Keegan LT, Karlen D, Chinazzi M, Davis JT, Mu K, Xiong X, Pastore Y Piontti A, Vespignani A, Srivastava A, Porebski P, Venkatramanan S, Adiga A, Lewis B, Klahn B, Outten J, Schlitt J, Corbett P, Telionis PA, Wang L, Peddireddy AS, Hurt B, Chen J, Vullikanti A, Marathe M, Healy JM, Slayton RB, Biggerstaff M, Johansson MA, Shea K, Lessler J. Modeling of future COVID-19 cases, hospitalizations, and deaths, by vaccination rates and nonpharmaceutical intervention scenarios-United States, april-september 2021. MMWR. Morbidity and Mortality Weekly Report. 2021;70:719–724. doi: 10.15585/mmwr.mm7019e3. PubMed DOI PMC

Bosse NI, Gruson H, Funk S, Abbott S. Scoringutils: utilities for scoring and assessing predictions. CRAN. 2023 https://github.com/epiforecasts/scoringutils

Bracher J, Wolffram D, Deuschel J, Görgen K, Ketterer J, SchienleM M. The German and Polish COVID-19 forecast hub. Github. 2020 https://github.com/KITmetricslab/covid19-forecast-hub-de

Bracher J, Ray EL, Gneiting T, Reich NG. Evaluating epidemic forecasts in an interval format. PLOS Computational Biology. 2021a;17:e1008618. doi: 10.1371/journal.pcbi.1008618. PubMed DOI PMC

Bracher J, Wolffram D, Deuschel J, Görgen K, Ketterer JL, Ullrich A, Abbott S, Barbarossa MV, Bertsimas D, Bhatia S, Bodych M, Bosse NI, Burgard JP, Castro L, Fairchild G, Fiedler J, Fuhrmann J, Funk S, Gambin A, Gogolewski K, Heyder S, Hotz T, Kheifetz Y, Kirsten H, Krueger T, Krymova E, Leithäuser N, Li ML, Meinke JH, Miasojedow B, Michaud IJ, Mohring J, Nouvellet P, Nowosielski JM, Ozanski T, Radwan M, Rakowski F, Scholz M, Soni S, Srivastava A, Gneiting T, Schienle M. National and Subnational Short-Term Forecasting of COVID-19 in Germany and Poland during Early 2021. medRxiv. 2021b doi: 10.1101/2021.11.05.21265810. PubMed DOI PMC

Bracher J, Wolffram D, Deuschel J, Görgen K, Ketterer JL, Ullrich A, Abbott S, Barbarossa MV, Bertsimas D, Bhatia S, Bodych M, Bosse NI, Burgard JP, Castro L, Fairchild G, Fuhrmann J, Funk S, Gogolewski K, Gu Q, Heyder S, Hotz T, Kheifetz Y, Kirsten H, Krueger T, Krymova E, Li ML, Meinke JH, Michaud IJ, Niedzielewski K, Ożański T, Rakowski F, Scholz M, Soni S, Srivastava A, Zieliński J, Zou D, Gneiting T, Schienle M, List of Contributors by Team A pre-registered short-term forecasting study of covid-19 in Germany and Poland during the second wave. Nature Communications. 2021c;12:5173. doi: 10.1038/s41467-021-25207-0. PubMed DOI PMC

Brooks L. Comparing ensemble approaches for short-term probabilistic COVID-19 forecasts in the U.S. 2020. [April 1, 2022]. https://forecasters.org/blog/2020/10/28/comparing-ensemble-approaches-for-short-term-probabilistic-covid-19-forecasts-in-the-u-s/

Buizza R. Introduction to the special issue on `` 25 years of ensemble forecasting.''. Quarterly Journal of the Royal Meteorological Society. 2019;145:1–11. doi: 10.1002/qj.3370. DOI

Castro M, Ares S, Cuesta JA, Manrubia S. The turning point and end of an expanding epidemic can not be precisely forecast. PNAS. 2020;117:26190–26196. doi: 10.1073/pnas.2007868117. PubMed DOI PMC

Català M, Pino D, Marchena M, Palacios P, Urdiales T, Cardona P-J, Alonso S, López-Codina D, Prats C, Alvarez-Lacalle E. Robust estimation of diagnostic rate and real incidence of COVID-19 for European policymakers. PLOS ONE. 2021;16:e0243701. doi: 10.1371/journal.pone.0243701. PubMed DOI PMC

CDC Coronavirus disease 2019. COVID-19. 2020 https://www.cdc.gov/coronavirus/2019-ncov/science/forecasting/forecasting.html

covid19-forecast-hub-europe Predictive performance of multi-model ensemble forecasts of covid-19 across European nations. Github. 2022 https://github.com/covid19-forecast-hub-europe/euro-hub-ensemble PubMed PMC

Cramer EY, Huang Y, Wang Y, Ray EL, Cornell M, Bracher J, Brennen A, Castero Rivadeneira AJ, Gerding A, House K, Jayawardena D, Kanji AH, Khandelwal A, Le K, Niemi J, Stark A, Shah A, Wattanchit N, Zorn MW, Reich NG. The United States COVID-19 Forecast Hub Dataset. medRxiv. 2021a doi: 10.1101/2021.11.04.21265886. PubMed DOI PMC

Cramer EY, Ray EL, Lopez VK, Bracher J, Brennen A, Castro Rivadeneira AJ, Gerding A, Gneiting T, House KH, Huang Y, Jayawardena D, Kanji AH, Khandelwal A, Le K, Mühlemann A, Niemi J, Shah A, Stark A, Wang Y, Wattanachit N, Zorn MW, Gu Y, Jain S, Bannur N, Deva A, Kulkarni M, Merugu S, Raval A, Shingi S, Tiwari A, White J, Abernethy NF, Woody S, Dahan M, Fox S, Gaither K, Lachmann M, Meyers LA, Scott JG, Tec M, Srivastava A, George GE, Cegan JC, Dettwiller ID, England WP, Farthing MW, Hunter RH, Lafferty B, Linkov I, Mayo ML, Parno MD, Rowland MA, Trump BD, Zhang-James Y, Chen S, Faraone SV, Hess J, Morley CP, Salekin A, Wang D, Corsetti SM, Baer TM, Eisenberg MC, Falb K, Huang Y, Martin ET, McCauley E, Myers RL, Schwarz T, Sheldon D, Gibson GC, Yu R, Gao L, Ma Y, Wu D, Yan X, Jin X, Wang YX, Chen Y, Guo L, Zhao Y, Gu Q, Chen J, Wang L, Xu P, Zhang W, Zou D, Biegel H, Lega J, McConnell S, Nagraj VP, Guertin SL, Hulme-Lowe C, Turner SD, Shi Y, Ban X, Walraven R, Hong QJ, Kong S, van de Walle A, Turtle JA, Ben-Nun M, Riley S, Riley P, Koyluoglu U, DesRoches D, Forli P, Hamory B, Kyriakides C, Leis H, Milliken J, Moloney M, Morgan J, Nirgudkar N, Ozcan G, Piwonka N, Ravi M, Schrader C, Shakhnovich E, Siegel D, Spatz R, Stiefeling C, Wilkinson B, Wong A, Cavany S, España G, Moore S, Oidtman R, Perkins A, Kraus D, Kraus A, Gao Z, Bian J, Cao W, Ferres JL, Li C, Liu TY, Xie X, Zhang S, Zheng S, Vespignani A, Chinazzi M, Davis JT, Mu K, y Piontti AP, Xiong X, Zheng A, Baek J, Farias V, Georgescu A, Levi R, Sinha D, Wilde J, Perakis G, Bennouna MA, Nze-Ndong D, Singhvi D, Spantidakis I, Thayaparan L, Tsiourvas A, Sarker A, Jadbabaie A, Shah D, Penna ND, Celi LA, Sundar S, Wolfinger R, Osthus D, Castro L, Fairchild G, Michaud I, Karlen D, Kinsey M, Mullany LC, Rainwater-Lovett K, Shin L, Tallaksen K, Wilson S, Lee EC, Dent J, Grantz KH, Hill AL, Kaminsky J, Kaminsky K, Keegan LT, Lauer SA, Lemaitre JC, Lessler J, Meredith HR, Perez-Saez J, Shah S, Smith CP, Truelove SA, Wills J, Marshall M, Gardner L, Nixon K, Burant JC, Wang L, Gao L, Gu Z, Kim M, Li X, Wang G, Wang Y, Yu S, Reiner RC, Barber R, Gakidou E, Hay SI, Lim S, Murray CJL, Pigott D, Gurung HL, Baccam P, Stage SA, Suchoski BT, Prakash BA, Adhikari B, Cui J, Rodríguez A, Tabassum A, Xie J, Keskinocak P, Asplund J, Baxter A, Oruc BE, Serban N, Arik SO, Dusenberry M, Epshteyn A, Kanal E, Le LT, Li CL, Pfister T, Sava D, Sinha R, Tsai T, Yoder N, Yoon J, Zhang L, Abbott S, Bosse NI, Funk S, Hellewell J, Meakin SR, Sherratt K, Zhou M, Kalantari R, Yamana TK, Pei S, Shaman J, Li ML, Bertsimas D, Lami OS, Soni S, Bouardi HT, Ayer T, Adee M, Chhatwal J, Dalgic OO, Ladd MA, Linas BP, Mueller P, Xiao J, Wang Y, Wang Q, Xie S, Zeng D, Green A, Bien J, Brooks L, Hu AJ, Jahja M, McDonald D, Narasimhan B, Politsch C, Rajanala S, Rumack A, Simon N, Tibshirani RJ, Tibshirani R, Ventura V, Wasserman L, O’Dea EB, Drake JM, Pagano R, Tran QT, Tung Ho LS, Huynh H, Walker JW, Slayton RB, Johansson MA, Biggerstaff M, Reich NG. Evaluation of Individual and Ensemble Probabilistic Forecasts of COVID-19 Mortality in the US. medRxiv. 2021b doi: 10.1101/2021.02.03.21250974. DOI

Cramer E, Wang SY, Reich NG, Hanna A, Niem J, House K, Huang YD. Reichlab/covid19-forecast-hub: release for zenodo, 20210816. Zenodo. 2021c doi: 10.5281/zenodo.5208210. DOI

Diebold FX, Mariano RS. Comparing predictive accuracy. Journal of Business & Economic Statistics. 1995;13:253–263. doi: 10.1080/07350015.1995.10524599. DOI

Dong E, Du H, Gardner L. An interactive web-based dashboard to track covid-19 in real time. The Lancet. Infectious Diseases. 2020;20:533–534. doi: 10.1016/S1473-3099(20)30120-1. PubMed DOI PMC

EpiForecasts Project: ECDC European COVID-19 forecast hub. 0.1Zoltar. 2021 https://www.zoltardata.com/project/238

European Centre for Disease Prevention and Control Threat assessment brief: implications for the EU/EEA on the spread of the SARS-cov-2 delta (B.1.617.2) variant of concern. 2021a. [April 1, 2022]. https://www.ecdc.europa.eu/en/publications-data/threat-assessment-emergence-and-impact-sars-cov-2-delta-variant

European Centre for Disease Prevention and Control Interim guidance on the benefits of full vaccination against COVID-19 for transmission and implications for non-pharmaceutical interventions. 2021b. [April 1, 2022]. https://www.ecdc.europa.eu/en/publications-data/interim-guidance-benefits-full-vaccination-against-covid-19-transmission

European Centre for Disease Prevention and Control Forecasting COVID-19 cases and deaths in Europe-new hub will support European pandemic planning. 2021c. [April 1, 2022]. https://www.ecdc.europa.eu/en/news-events/forecasting-covid-19-cases-and-deaths-europe-new-hub

European Centre for Disease Prevention and Control Weekly threats reports (CDTR) 2022a. [April 1, 2023]. https://www.ecdc.europa.eu/en/publications-and-data/monitoring/weekly-threats-reports

European Centre for Disease Prevention and Control Assessment of the further spread and potential impact of the SARS-CoV-2 Omicron variant of concern in the EU/EEA, 19th update. 2022b. [April 1, 2022]. https://www.ecdc.europa.eu/en/publications-data/covid-19-omicron-risk-assessment-further-emergence-and-potential-impact

European Centre for Disease Prevention and Control Overview of the implementation of COVID-19 vaccination strategies and deployment plans in the EU/EEA. 2023. [April 1, 2023]. https://www.ecdc.europa.eu/en/publications-data/overview-implementation-covid-19-vaccination-strategies-and-deployment-plans

European Covid-19 Forecast Hub Community. 2023a. [April 1, 2023]. https://covid19forecasthub.eu/community.html

European Covid-19 Forecast Hub European Covid-19 Forecast Hub. 2023b. [April 1, 2023]. https://covid19forecasthub.eu/index.html

European Covid-19 Forecast Hub Covid19-forecast-hub-europe. 9d13832Github. 2023c https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe

European Covid-19 Forecast Hub Covid19-forecast-hub-europe, 2021. 9d13832Github. 2023d https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe

Friedman J, Liu P, Troeger CE, Carter A, Reiner RC, Barber RM, Collins J, Lim SS, Pigott DM, Vos T, Hay SI, Murray CJL, Gakidou E. Predictive performance of international COVID-19 mortality forecasting models. Nature Communications. 2021;12:2609. doi: 10.1038/s41467-021-22457-w. PubMed DOI PMC

Funk S, Abbott S, Atkins BD, Baguelin M, Baillie JK, Birrell P, Blake J, Bosse NI, Burton J, Carruthers J, Davies NG, De Angelis D, Dyson L, Edmunds WJ, Eggo RM, Ferguson NM, Gaythorpe K, Gorsich E, Guyver-Fletcher G, Hellewell J, Hill EM, Holmes A, House TA, Jewell C, Jit M, Jombart T, Joshi I, Keeling MJ, Kendall E, Knock ES, Kucharski AJ, Lythgoe KA, Meakin SR, Munday JD, Openshaw PJM, Overton CE, Pagani F, Pearson J, Perez-Guzman PN, Pellis L, Scarabel F, Semple MG, Sherratt K, Tang M, Tildesley MJ, Van Leeuwen E, Whittles LK, CMMID COVID-19 Working Group. Imperial College COVID-19 Response Team. ISARIC4C Investigators Short-Term Forecasts to Inform the Response to the Covid-19 Epidemic in the UK. medRxiv. 2020 doi: 10.1101/2020.11.11.20220962. DOI

Genest C. Vincentization revisited. The Annals of Statistics. 1992;20:1137–1142. doi: 10.1214/aos/1176348676. DOI

Gneiting T, Raftery AE. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association. 2007;102:359–378. doi: 10.1198/016214506000001437. DOI

Harrell FE, Davis CE. A new distribution-free quantile estimator. Biometrika. 1982;69:635–640. doi: 10.1093/biomet/69.3.635. DOI

James LP, Salomon JA, Buckee CO, Menzies NA. The use and misuse of mathematical modeling for infectious disease policymaking: lessons for the COVID-19 pandemic. Medical Decision Making. 2021;41:379–385. doi: 10.1177/0272989X21990391. PubMed DOI PMC

Jin R. The lag between daily reported covid-19 cases and deaths and its relationship to age. Journal of Public Health Research. 2021;10:2049. doi: 10.4081/jphr.2021.2049. PubMed DOI PMC

Johansson MA, Apfeldorf KM, Dobson S, Devita J, Buczak AL, Baugher B, Moniz LJ, Bagley T, Babin SM, Guven E, Yamana TK, Shaman J, Moschou T, Lothian N, Lane A, Osborne G, Jiang G, Brooks LC, Farrow DC, Hyun S, Tibshirani RJ, Rosenfeld R, Lessler J, Reich NG, Cummings DAT, Lauer SA, Moore SM, Clapham HE, Lowe R, Bailey TC, García-Díez M, Carvalho MS, Rodó X, Sardar T, Paul R, Ray EL, Sakrejda K, Brown AC, Meng X, Osoba O, Vardavas R, Manheim D, Moore M, Rao DM, Porco TC, Ackley S, Liu F, Worden L, Convertino M, Liu Y, Reddy A, Ortiz E, Rivero J, Brito H, Juarrero A, Johnson LR, Gramacy RB, Cohen JM, Mordecai EA, Murdock CC, Rohr JR, Ryan SJ, Stewart-Ibarra AM, Weikel DP, Jutla A, Khan R, Poultney M, Colwell RR, Rivera-García B, Barker CM, Bell JE, Biggerstaff M, Swerdlow D, Mier-Y-Teran-Romero L, Forshey BM, Trtanj J, Asher J, Clay M, Margolis HS, Hebbeler AM, George D, Chretien JP. An open challenge to advance probabilistic forecasting for dengue epidemics. PNAS. 2019;116:24268–24274. doi: 10.1073/pnas.1909865116. PubMed DOI PMC

Moran KR, Fairchild G, Generous N, Hickmann K, Osthus D, Priedhorsky R, Hyman J, Del Valle SY. Epidemic forecasting is messier than weather forecasting: the role of human behavior and Internet data streams in epidemic forecast. The Journal of Infectious Diseases. 2016;214:S404–S408. doi: 10.1093/infdis/jiw375. PubMed DOI PMC

Ray EL, Wattanachit N, Niemi J, Kanji AH, House K, Cramer EY, Bracher J, Zheng A, Yamana TK, Xiong X, Woody S, Wang Y, Wang L, Walraven RL, Tomar V, Sherratt K, Sheldon D, Reiner RC, Prakash BA, Osthus D, Li ML, Lee EC, Koyluoglu U, Keskinocak P, Gu Y, Gu Q, George GE, España G, Corsetti S, Chhatwal J, Cavany S, Biegel H, Ben-Nun M, Walker J, Slayton R, Lopez V, Biggerstaff M, Johansson MA, Reich NG. Ensemble Forecasts of Coronavirus Disease 2019 (COVID-19) in the U.S. medRxiv. 2020 doi: 10.1101/2020.08.19.20177493. DOI

Ray EL, Brooks LC, Bien J, Biggerstaff M, Bosse NI, Bracher J, Cramer EY, Funk S, Gerding A, Johansson MA, Rumack A, Wang Y, Zorn M, Tibshirani RJ, Reich NG. Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States. International Journal of Forecasting. 2022;1:005. doi: 10.1016/j.ijforecast.2022.06.005. PubMed DOI PMC

Reich NG, Brooks LC, Fox SJ, Kandula S, McGowan CJ, Moore E, Osthus D, Ray EL, Tushar A, Yamana TK, Biggerstaff M, Johansson MA, Rosenfeld R, Shaman J. A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States. PNAS. 2019a;116:3146–3154. doi: 10.1073/pnas.1812594116. PubMed DOI PMC

Reich NG, McGowan CJ, Yamana TK, Tushar A, Ray EL, Osthus D, Kandula S, Brooks LC, Crawford-Crudell W, Gibson GC, Moore E, Silva R, Biggerstaff M, Johansson MA, Rosenfeld R, Shaman J. Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S. PLOS Computational Biology. 2019b;15:e1007486. doi: 10.1371/journal.pcbi.1007486. PubMed DOI PMC

Reich NG, Cornell M, Ray EL, House K, Le K. The zoltar forecast archive, a tool to standardize and store interdisciplinary prediction research. Scientific Data. 2021;8:59. doi: 10.1038/s41597-021-00839-5. PubMed DOI PMC

Sherratt K. 2022. European covid-19 forecast hub. Zenodo. DOI

Taylor JW, Taylor KS. Combining probabilistic forecasts of covid-19 mortality in the United States. European Journal of Operational Research. 2023;304:25–41. doi: 10.1016/j.ejor.2021.06.044. PubMed DOI PMC

Van Basshuysen P, White L, Khosrowi D, Frisch M. Three ways in which pandemic models may perform a pandemic. Erasmus Journal for Philosophy and Economics. 2021;14:10–127. doi: 10.23941/ejpe.v14i1.582. DOI

Viboud C, Sun K, Gaffey R, Ajelli M, Fumanelli L, Merler S, Zhang Q, Chowell G, Simonsen L, Vespignani A, RAPIDD Ebola Forecasting Challenge group The RAPIDD Ebola forecasting challenge: synthesis and lessons learnt. Epidemics. 2018;22:13–21. doi: 10.1016/j.epidem.2017.08.002. PubMed DOI PMC

Wang SY, Stark A, Ray E, Bosse N, Reich NG, Sherratt K, Shah A. 2021. Reichlab/covidhubutils: Repository release for zenodo. Zenodo. DOI

Zelner J, Riou J, Etzioni R, Gelman A. Accounting for uncertainty during a pandemic. Patterns. 2021;2:100310. doi: 10.1016/j.patter.2021.100310. PubMed DOI PMC

Using real-time ascertainment rate estimate from infection and hospitalization dataset for modeling the spread of infectious disease: COVID-19 case study in the Czech Republic

Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations