Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks.

Advanced Data Analytics Metron Inc Reston VA 20190

Catalog Data Science Walmart Inc Sunnyvale CA 94085

Centre for Mathematical Modelling of Infectious Diseases London School of Hygiene and Tropical Medicine WC1E 7HT London United Kingdom

Chair of Econometrics and Statistics Karlsruhe Institute of Technology 76185 Karlsruhe Germany

College of Computing Georgia Institute of Technology Atlanta GA 30308

College of Information and Computer Sciences University of Massachusetts Amherst MA 01003

Computational Statistics Group Heidelberg Institute for Theoretical Studies 69118 Heidelberg Germany

Construx Bellevue WA 98004

Core Consultant Group Oliver Wyman New York NY 10036

COVID 19 Response Centers for Disease Control and Prevention; Atlanta GA 30333

Department of Biological Sciences University of Notre Dame Notre Dame IN 46556

Department of Biomedical Data Sciences Stanford University Stanford CA 94305

Department of Biostatistics and Epidemiology University of Massachusetts Amherst MA 01003

Department of Biostatistics Columbia University New York NY 10032

Department of Biostatistics University of North Carolina at Chapel Hill Chapel Hill NC 27599

Department of Biostatistics University of Washington Seattle WA 98195

Department of Civil and Environmental Engineering University of Washington Seattle WA 98195

Department of Civil and Systems Engineering Johns Hopkins University Baltimore MD 21218

Department of Complex Systems University of Michigan Ann Arbor MI 48109

Department of Computer Science and Engineering University of California San Diego CA 92093

Department of Computer Science University of California Los Angeles CA 90095

Department of Computer Science University of California Santa Barbara CA 93106

Department of Computer Science University of Iowa Iowa City IA 52242

Department of Computer Science Virginia Tech Falls Church VA 22043

Department of Ecology and Evolution University of Chicago Chicago IL 60637

Department of Electrical and Computer Engineering The University of Texas at Austin Austin TX 78712

Department of Electrical Engineering and Computer Science Syracuse University Syracuse NY 13207

Department of Environmental Health Sciences Columbia University New York NY 10032

Department of Epidemiology Johns Hopkins Bloomberg School of Public Health Baltimore MD 21215

Department of Finance Iowa State University Ames IA 50011

Department of Health Policy and Management Harvard University Cambridge MA 02138

Department of Information Risk and Operations Management University of Texas at Austin Austin TX 78712

Department of Integrative Biology University of Texas at Austin Austin TX 78712

Department of International Health Johns Hopkins Bloomberg School of Public Health Baltimore MD 21231

Department of Materials Science and Engineering Rensselaer Polytechnic Institute Troy NY 12309

Department of Mathematics and Statistics Dalhousie University Halifax NS B3H 4R2 Canada

Department of Mathematics and Statistics Masaryk University 61137 Brno Czech Republic

Department of Mathematics College of William and Mary Williamsburg VA 23187

Department of Mathematics University of Arizona Tucson AZ 85721

Department of Mathematics University of Michigan Ann Arbor MI 48109

Department of Medicine Section of Infectious Diseases Boston University School of Medicine Boston MA 02118

Department of Physics and Astronomy University of Victoria Victoria BC V8W 2Y2 Canada

Department of Physics Trinity University San Antonio TX 78212

Department of Physics University of Michigan Ann Arbor MI 48109

Department of Psychiatry and Behavioral Sciences State University of New York Upstate Medical University Syracuse NY 13210

Department of Psychiatry Columbia University New York NY 10032

Department of Public Health and Preventive Medicine State University of New York Upstate Medical University Syracuse NY 13210

Department of Statistics and Data Sciences University of Texas at Austin Austin TX 78712

Department of Statistics Carnegie Mellon University Pittsburgh PA 15213

Department of Statistics Iowa State University Ames IA 50011

Department of Statistics Stanford University Stanford CA 94305

Department of Statistics University of British Columbia Vancouver BC V6T 1Z4 Canada

Department of Statistics University of Virginia Charlottesville VA 22904

Department of Statistics University of Washington Seattle WA 98185

Division of Epidemiology Department of Internal Medicine University of Utah Salt Lake City UT 84108

Emerging Technologies IEM Inc Baton Rouge LA 70809

Emerging Technologies IEM Inc Bel Air MD 21015

Financial Services Oliver Wyman Digital Toronto ON Canada M5J 0A1

Financial Services Oliver Wyman London UK W1U 8EW

Financial Services Oliver Wyman New York NY 10036

Google Cloud Sunnyvale CA 94089

H Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta GA 30332

Halıcıoğlu Data Science Institute University of California San Diego CA 92093

Health and Life Sciences Oliver Wyman Boston MA 02110

Health and Life Sciences Oliver Wyman New York NY 10036

Health Economic Modeling Value Analytics Labs 34776 İstanbul Turkey

Infectious Disease Group Predictive Science Inc San Diego CA 92121

Information Systems and Modeling Group Los Alamos National Laboratory Los Alamos NM 87545

Institute for Computational Medicine Johns Hopkins Bloomberg School of Public Health Baltimore MD 21218

Institute for Data Systems and Society Massachusetts Institute of Technology Cambridge MA 02139

Institute for Health Metrics and Evaluation University of Washington Seattle WA 98195

Institute for Scientific Interchange Foundation Turin 10133 Italy

Institute of Mathematical Statistics and Actuarial Science University of Bern CH 3012 Bern Switzerland

Institute of Stochastics Karlsruhe Institute of Technology 69118 Karlsruhe Germany

International Vaccine Access Center Johns Hopkins University Baltimore MD 21231

IQT Labs In Q Tel Waltham MA 02451

Jilin University Changchun City Jilin Province 130012 People's Republic of China

JMP Life Sciences SAS Institute Inc Cary NC 27513

Johns Hopkins University Applied Physics Laboratory Laurel MD 20723

Khoury College of Computer Sciences Northeastern University Boston MA 02115

Laboratory for Computational Physiology Massachusetts Institute of Technology Cambridge MA 02139

Laboratory for the Modeling of Biological and Socio technical Systems Northeastern University Boston MA 02115

Laboratory of Ecohydrology School of Architecture Civil and Environmental Engineering École Polytechnique Fédérale de Lausanne 1015 Lausanne Switzerland

Machine Learning Department Carnegie Mellon University Pittsburgh PA 15213

Manhasset Secondary School Manhasset NY 11030

Marshall School of Business Department of Data Sciences and Operations University of Southern California Los Angeles CA 90089

Massachusetts Institute of Technology Cambridge MA 02142

McCombs School of Business The University of Texas at Austin Austin TX 78712

Mechatronics Embedded Systems and Automation Lab Department of Mechanical Engineering University of California Merced CA 95301

Medical Research Council Centre for Global Infectious Disease Analysis Department of Infectious Disease Epidemiology School of Public Health Imperial College W2 1PG London United Kingdom

Microsoft Redmond WA 98029

Ming Hsieh Department of Computer and Electrical Engineering University of Southern California Los Angeles CA 90089

Odum School of Ecology University of Georgia Athens GA 30602

Oliver Wyman Digital Oliver Wyman Boston MA 02110

Oliver Wyman Digital Oliver Wyman New York NY 10036

Oliver Wyman Digital Oliver Wyman Sao Paolo Brazil 04711 904

Operations Research Center Massachusetts Institute of Technology; Cambridge MA 02139

Physical Sciences Division TRIUMF Vancouver BC V8W 2Y2 Canada

Quality Assurance and Data Science Signature Science LLC Austin TX 78759

Quality Assurance and Data Science Signature Science LLC Charlottesville VA 22911

Radiology Institute for Technology Assessment Massachusetts General Hospital Boston MA 02114

River Hill High School Clarksville MD 21029

Santa Fe Institute Santa Fe NM 87501

School for Engineering of Matter Transport and Energy Arizona State University Tempe AZ 85287

School of Engineering Brown University Providence RI 02912

School of Mathematical and Statistical Sciences Clemson University Clemson SC 29634

School of Medicine State University of New York Upstate Medical University Syracuse NY 13210

School of Public Health and Health Sciences University of Massachusetts Amherst MA 01003

School of Public Health Department of Epidemiology University of Michigan Ann Arbor MI 48109

Sloan School of Management Massachusetts Institute of Technology Cambridge MA 02142

Statistical Sciences Group Los Alamos National Laboratory Los Alamos NM 87545

Technology Operations and Statistics group Stern School of Business New York University New York NY 10012

Texas Advanced Computing Center Austin TX 78758

Unaffiliated Amsterdam The Netherlands

Unaffiliated Baltimore MD 21205

Unaffiliated Davis CA 95616

Unaffiliated New York NY 10016

Unaffiliated San Francisco CA 94122

Unaffiliated Tucson AZ 85704

University of Science and Technology of China Heifei Anhui 230027 People's Republic of China

University of Washington Seattle WA 98109

US Army Engineer Research and Development Center Concord MA 01742

US Army Engineer Research and Development Center Hanover NH 03755

US Army Engineer Research and Development Center Vicksburg MS 39180

Virtual Power System Inc Milpitas CA 95035

Wadhwani Institute of Artificial Intelligence Andheri East Mumbai 400093 India

Winship Cancer Institute Emory University Medical School Atlanta GA 30322

Erratum v

Proc Natl Acad Sci U S A. 2023 Apr 11;120(15):e2304076120. doi: 10.1073/pnas.2304076120. PubMed

Zobrazit více v PubMed

Davies S. E., Youde J. R., The Politics of Surveillance and Response to Disease Outbreaks: The New Frontier for States and Non-state Actors (Routledge, 2016).

Polonsky J. A., et al. , Outbreak analytics: A developing data science for informing the response to emerging pathogens. Philos. Trans. R. Soc. Lond. B Biol. Sci. 374, 20180276 (2019). PubMed PMC

Lutz C. S., et al. , Applying infectious disease forecasting to public health: A path forward using influenza forecasting examples. BMC Public Health 19, 1659 (2019). PubMed PMC

Cramer E., et al. , COVID-19 Forecast Hub: 4 December 2020 snapshot. https://zenodo.org/record/4305938#.Yf1TQOrMI2x (Accessed 11 December 2020).

CDC, COVID-19 Forecasting and Mathematical Modeling. Centers for Disease Control and Prevention. https://www.cdc.gov/coronavirus/2019-ncov/science/forecasting/mathematical-modeling.html (Accessed 2 March 2022).

Bates J. M., Granger C. W. J., The combination of forecasts. J. Oper. Res. Soc. 20, 451–468 (1969).

Krishnamurti T. N., et al. , Improved weather and seasonal climate forecasts from multimodel superensemble. Science 285, 1548–1550 (1999). PubMed

Gneiting T., Raftery A. E., Atmospheric science. Weather forecasting with ensemble methods. Science 310, 248–249 (2005). PubMed

Leutbecher M., Palmer T. N., Ensemble forecasting. J. Comput. Phys. 227, 3515–3539 (2008).

Polikar R., Ensemble based systems in decision making. IEEE Circuits Syst. Mag. 6, 21–45 (2006).

Lakshminarayanan B., Pritzel A., Blundell C., Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv [Preprint] (2017). https://arxiv.org/abs/1612.01474 (Accessed 24 December 2020).

McGowan C. J., et al. ; Influenza Forecasting Working Group, Collaborative efforts to forecast seasonal influenza in the United States, 2015-2016. Sci. Rep. 9, 683 (2019). PubMed PMC

Reich N. G., et al. , A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States. Proc. Natl. Acad. Sci. U.S.A. 116, 3146–3154 (2019). PubMed PMC

Reich N. G., et al. , Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S. PLOS Comput. Biol. 15, e1007486 (2019). PubMed PMC

Viboud C., et al. ; RAPIDD Ebola Forecasting Challenge group, The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt. Epidemics 22, 13–21 (2018). PubMed PMC

Johansson M. A., et al. , An open challenge to advance probabilistic forecasting for dengue epidemics. Proc. Natl. Acad. Sci. U.S.A. 116, 24268–24274 (2019). PubMed PMC

Funk S., et al. , Short-term forecasts to inform the response to the Covid-19 epidemic in the UK. medRxiv [Preprint] (2020). https://www.medrxiv.org/content/10.1101/2020.11.11.20220962v2 (Accessed 2 December 2020). DOI

Taylor K. S., Taylor J. W., A comparison of aggregation methods for probabilistic forecasts of COVID-19 mortality in the United States. arXiv [Preprint] (2020). https://arxiv.org/abs/2007.11103 (Accessed 2 December 2020).

Moran K. R., et al. , Epidemic forecasting is messier than weather forecasting: The role of human behavior and internet data streams in epidemic forecast. J. Infect. Dis. 214 (suppl. 4), S404–S408 (2016). PubMed PMC

Bracher J., Ray E. L., Gneiting T., Reich N. G., Evaluating epidemic forecasts in an interval format. PLOS Comput. Biol. 17, e1008618 (2021). PubMed PMC

Lerch S., Thorarinsdottir T. L., Ravazzolo F., Gneiting T., Forecaster’s dilemma: Extreme events and forecast evaluation. SSO Schweiz. Monatsschr. Zahnheilkd. 32, 106–127 (2017).

Bracher J., et al. ; List of Contributors by Team, A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave. Nat. Commun. 12, 5173 (2021). PubMed PMC

McDonald D. J., et al. , Can auxiliary indicators improve COVID-19 forecasting and hotspot prediction? bioRxiv [Preprint] (2021). 10.1101/2021.06.22.21259346. PubMed DOI PMC

Dong E., Du H., Gardner L., An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect. Dis. 20, 533–534 (2020). PubMed PMC

Department of Health, NM-IBIS - MMWR week description and corresponding calendar dates (2006–2025). https://ibis.health.state.nm.us/resource/MMWRWeekCalendar.html (Accessed 13 January 2021).

Cramer E. Y., et al. , The United States COVID-19 Forecast Hub hub dataset. medRxiv [Preprint] (2021). 10.1101/2021.11.04.21265886 (Accessed 4 December 2021). DOI

Ray E. L., et al. , Ensemble forecasts of coronavirus disease 2019 (COVID-19) in the U.S. medRxiv [Preprint] (2020). https://www.medrxiv.org/content/10.1101/2020.08.19.20177493v1 (Accessed 2 December 2020). DOI

Brooks L. C., et al. , Comparing Ensemble Approaches for Short-term Probabilistic COVID-19 Forecasts in the U.S. (International Institute of Forecasters, 2020).

Ray E. L., et al. , Challenges in Training Ensembles to Forecast COVID-19 Cases and Deaths in the United States (International Institute of Forecasters, 2021).

Gneiting T., Raftery A. E., Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc. 102, 359–378 (2007).

Soloman S. R., Sawilowsky S. S., Impact of rank-based normalizing transformations on the accuracy of test scores. J. Mod. Appl. Stat. Methods 8, 448–462 (2009).

Wu S., Crestani F., Bi Y., “Evaluating score normalization methods in data fusion” in Information Retrieval Technology, Lecture notes in computer science., (Springer Berlin Heidelberg, 2006), pp. 642–648.

Renda M. E., Straccia U., “Web metasearch: Rank vs. score based rank aggregation methods” in Proceedings of the 2003 ACM Symposium on Applied Computing, SAC ’03. (Association for Computing Machinery, 2003), pp. 841–846.

E. Y. Cramer et al.., COVID-19 Forecast Hub. GitHub. https://github.com/reichlab/covid19-forecast-hub/tree/master/data-processed. Accessed 17 November 2021.

Reich N. G., Cornell M., Ray E. L., House K., Le K., The Zoltar forecast archive, a tool to standardize and store interdisciplinary prediction research. Sci. Data 8, 59 (2021). PubMed PMC

R Core Team, R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, 2020).

Pollett S., et al. , Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines. PLoS Med. 18, e1003793 (2021). PubMed PMC