Learning to Forecast: The Probabilistic Time Series Forecasting Challenge

References

• Bojer, C. S., and Meldgaard, J. P. (2021), “Kaggle Forecasting Competitions: An Overlooked Learning Opportunity,” International Journal of Forecasting, 37, 587–603. DOI: 10.1016/j.ijforecast.2020.07.007.
   Web of Science ®Google Scholar
• Bowles, C., Friz, R., Genre, V., Kenny, G., Meyler, A., and Rautanen, T. (2007), “The ECB Survey of Professional Forecasters (SPF) - A Review after Eight Years’ Experience,” ECB occasional paper no. 59.
   Google Scholar
• Bracher, J., Koster, N., Krüger, F., Lerch, S., and Wolffram, D. (2021a), “Preregistration: Probabilistic Time Series Forecasting Challenge, Karlsruhe Institute of Technology,” https://osf.io/npwcv/ (last accessed: October 27, 2022).
   Google Scholar
• Bracher, J., Wolffram, D., Deuschel, J., Görgen, K., Ketterer, J., Ullrich, A., Abbott, S., Barbarossa, M., Bertsimas, D., Bhatia, S., Bodych, M., Bosse, N., Burgard, J., Fuhrmann, J., Funk, S., Gogolewski, K., Gu, Q., Heyder, S., Hotz, T., Kheifetz, Y., Kirsten, H., Krueger, T., Krymova, E., Li, M., Meinke, J., Niedzielewski, K., Ożański, T., Rakowski, F., Scholz, M., Soni, S., Srivastava, A., Zieliński, J., Zou, D., Gneiting, T., and Schienle, M. (2021b), “A Pre-registered Short-Term Forecasting Study of COVID-19 in Germany and Poland during the Second Wave,” Nature Communications, 12, 5173. DOI: 10.1038/s41467-021-25207-0.
   PubMed Web of Science ®Google Scholar
• Brockwell, P. J., and Davis, R. A. (2016), Introduction to Time Series and Forecasting (3rd ed.), Cham: Springer.
   Google Scholar
• Çetinkaya-Rundel, M., and Ellison, V. (2021), “A Fresh Look at Introductory Data Science,” Journal of Statistics and Data Science Education, 29, S16–S26. DOI: 10.1080/10691898.2020.1804497.
   Web of Science ®Google Scholar
• Clements, M. P. (2018), “Are Macroeconomic Density Forecasts Informative?” International Journal of Forecasting, 34, 181–198. DOI: 10.1016/j.ijforecast.2017.10.004.
   Web of Science ®Google Scholar
• Cramer, E. Y., Ray, E. L., Lopez, V. K., Bracher, J., Brennen, A., Castro, A. J., et al. (2022), “Evaluation of Individual and Ensemble Probabilistic Forecasts of COVID-19 Mortality in the United States,” Proceedings of the National Academy of Sciences, 119, e2113561119. DOI: 10.1073/pnas.2113561119.
   PubMed Web of Science ®Google Scholar
• Croushore, D. (2006), “Forecasting with Real-Time Macroeconomic Data,” in Handbook of Economic Forecasting (Vol. 1), eds. G. Elliott, C. W. Granger, and A. Timmermann, pp. 961–982, Amsterdam: Elsevier.
   Google Scholar
• Croushore, D., and Stark, T. (2019), “Fifty Years of the Survey of Professional Forecasters,” Federal Reserve Bank of Philadelphia Economic Insights, Q4, 1–11.
   Google Scholar
• Dicheva, D., Dichev, C., Agre, G., and Angelova, G. (2015), “Gamification in Education: A Systematic Mapping Study,” Journal of Educational Technology & Society, 18, 75–88.
   Web of Science ®Google Scholar
• Epstein, E. S. (1969), “A Scoring System for Probability Forecasts of Ranked Categories,” Journal of Applied Meteorology, 8, 985–987. DOI: 10.1175/1520-0450(1969)008<0985:ASSFPF>2.0.CO;2.
   Google Scholar
• Faust, J., and Wright, J. H. (2013), “Forecasting Inflation,” in Handbook of Economic Forecasting, volume 2 of Handbook of Economic Forecasting, eds. G. Elliott and A. Timmermann, pp. 2–56, Amsterdam: Elsevier.
   Google Scholar
• Frongillo, R., Gomez, R., Thilagar, A., and Waggoner, B. (2021), “Efficient Competitions and Online Learning with Strategic Forecasters,” in Proceedings of the 22nd ACM Conference on Economics and Computation, pp. 479–496. DOI: 10.1145/3465456.3467635.
   Google Scholar
• Glas, A., and Hartmann, M. (2022), “Uncertainty Measures from Partially Rounded Probabilistic Forecast Surveys,” Quantitative Economics, 13, 979–1022. DOI: 10.3982/QE1703.
   Web of Science ®Google Scholar
• Gneiting, T., and Katzfuss, M. (2014), “Probabilistic Forecasting,” Annual Review of Statistics and Its Application, 1, 125–151. DOI: 10.1146/annurev-statistics-062713-085831.
   Web of Science ®Google Scholar
• Gneiting, T., and Raftery, A. E. (2005), “Weather Forecasting with Ensemble Methods,” Science, 310, 248–249. DOI: 10.1126/science.1115255.
   PubMed Web of Science ®Google Scholar
• Gneiting, T., and Raftery, A. E. (2007), “Strictly Proper Scoring Rules, Prediction, and Estimation,” Journal of the American Statistical Association, 102, 359–378.
   Web of Science ®Google Scholar
• Gneiting, T., Raftery, A. E., Westveld III, A. H., and Goldman, T. (2005), “Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation,” Monthly Weather Review, 133, 1098–1118. DOI: 10.1175/MWR2904.1.
   Web of Science ®Google Scholar
• Gneiting, T., Wolffram, D., Resin, J., Kraus, K., Bracher, J., Dimitriadis, T., Hagenmeyer, V., Jordan, A. I., Lerch, S., Phipps, K., and Schienle, M. (2023), “Model Diagnostics and Forecast Evaluation for Quantiles,” Annual Review of Statistics and Its Application, 10, 597–621. DOI: 10.1146/annurev-statistics-032921-020240.
   Web of Science ®Google Scholar
• Hamari, J., Koivisto, J., and Sarsa, H. (2014), “Does Gamification Work? – A Literature Review of Empirical Studies on Gamification,” in 2014 47th Hawaii International Conference on System Sciences, pp. 3025–3034.
   Google Scholar
• Hastie, T., Tibshirani, R., and Friedman, J. H. (2009), The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.), New York: Springer
   Google Scholar
• Hong, T., Pinson, P., Fan, S., Zareipour, H., Troccoli, A., and Hyndman, R. J. (2016), “Probabilistic Energy Forecasting: Global Energy Forecasting Competition 2014 and Beyond,” International Journal of Forecasting, 32, 896–913. DOI: 10.1016/j.ijforecast.2016.02.001.
   Web of Science ®Google Scholar
• Hwang, J., Orenstein, P., Cohen, J., Pfeiffer, K., and Mackey, L. (2019), “Improving Subseasonal Forecasting in the Western US with Machine Learning,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2325–2335.
   Google Scholar
• Hyndman, R. J. (2020), “A Brief History of Forecasting Competitions,” International Journal of Forecasting, 36, 7–14. DOI: 10.1016/j.ijforecast.2019.03.015.
   Web of Science ®Google Scholar
• Hyndman, R. J., and Athanasopoulos, G. (2021), Forecasting: Principles and Practice (3rd ed.), OTexts. Available at https://otexts.com/fpp3/ (last accessed: March 16, 2023).
   Google Scholar
• James, G., Witten, D., Hastie, T., and Tibshirani, R. (2021). An Introduction to Statistical Learning (2nd ed.), New York: Springer.
   Google Scholar
• Krüger, F. (2017), “Survey-based Forecast Distributions for Euro Area Growth and Inflation: Ensembles versus Histograms,” Empirical Economics, 53, 235–246. DOI: 10.1007/s00181-017-1228-3.
   Web of Science ®Google Scholar
• Krüger, F., and Ziegel, J. F. (2021), “Generic Conditions for Forecast Dominance,” Journal of Business & Economic Statistics, 39, 972–983. DOI: 10.1080/07350015.2020.1741376.
   Web of Science ®Google Scholar
• Laio, F., and Tamea, S. (2007), “Verification Tools for Probabilistic Forecasts of Continuous Hydrological Variables,” Hydrology and Earth System Sciences, 11, 1267–1277. DOI: 10.5194/hess-11-1267-2007.
   Web of Science ®Google Scholar
• Lichtendahl, K. C., Grushka-Cockayne, Y., and Winkler, R. L. (2013), “Is it Better to Average Probabilities or Quantiles?” Management Science, 59, 1594–1611. DOI: 10.1287/mnsc.1120.1667.
   Web of Science ®Google Scholar
• M Open Forecasting Center. (2023a), M6 Forecasting Competition. Nicosia, Cyprus. Available at https://mofc.unic.ac.cy/the-m6-competition/ (last accessed: March 15, 2023).
   Google Scholar
• M Open Forecasting Center. (2023b). M6 Forecasting Competition: Guidelines. Nicosia, Cyprus. Available at https://mofc.unic.ac.cy/wp-content/uploads/2022/02/M6-Guidelines-1.pdf (last accessed: March 15, 2023).
   Google Scholar
• Makridakis, S., Fry, C., Petropoulos, F., and Spiliotis, E. (2022), “The Future of Forecasting Competitions: Design Attributes and Principles,” INFORMS Journal on Data Science, 1, 96–113. DOI: 10.1287/ijds.2021.0003.
   Google Scholar
• Messner, J. W., Mayr, G. J., and Zeileis, A. (2016), “Heteroscedastic Censored and Truncated Regression with crch,” R Journal, 8, 173–181. DOI: 10.32614/RJ-2016-012.
   Web of Science ®Google Scholar
• Petropoulos, F., Apiletti, D., Assimakopoulos, V., Babai, M. Z., Barrow, D. K., Taieb, S. B., Bergmeir, C., Bessa, R. J., Bijak, J., Boylan, J. E., et al. (2022), “Forecasting: Theory and Practice,” International Journal of Forecasting, 38, 705–871. DOI: 10.1016/j.ijforecast.2021.11.001.
   Web of Science ®Google Scholar
• Pfeifer, P. E., Grushka-Cockayne, Y., and Lichtendahl Jr, K. C. (2014), “The Promise of Prediction Contests,” The American Statistician, 68, 264–270. DOI: 10.1080/00031305.2014.937545.
   Web of Science ®Google Scholar
• Plotly Technologies Inc. (2015), plotly: Collaborative Data Science, available at https://plot.ly (last accessed: November 28, 2022).
   Google Scholar
• Pohle, M.-O. (2020), “The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation,” Preprint, arxiv:2005.01835.
   Google Scholar
• Posit Software, PBC. (2023). “Shiny,” available at https://shiny.rstudio.com (last accessed: April 10, 2023).
   Google Scholar
• R Core Team. (2022), R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. Available at https://www.R-project.org/ (last accessed: November 28, 2022).
   Google Scholar
• Raschka, S. (2022), “Deeper Learning by Doing: Integrating Hands-on Research Projects into a Machine Learning Course,” in Proceedings of the Second Teaching Machine Learning and Artificial Intelligence Workshop, volume 170 of Proceedings of Machine Learning Research, eds. K. M. Kinnaird, P. Steinbach, and O. Guhr, pp. 46–50. PMLR.
   Google Scholar
• Sherratt, K., Gruson, H., Grah, R., Johnson, H., Niehus, R., Prasse, B., et al. (2022), “Predictive Performance of Multi-Model Ensemble Forecasts of COVID-19 Across European Nations,” Preprint, medRxiv.
   Google Scholar
• Taylor, S. J., and Letham, B. (2018), “Forecasting at Scale,” The American Statistician, 72, 37–45. DOI: 10.1080/00031305.2017.1380080.
   Web of Science ®Google Scholar
• Timmermann, A. (2006), “Forecast Combinations,” in Handbook of Economic Forecasting (Vol. 1), eds. G. Elliott, C. Granger, and A. Timmermann, pp. 135–196, Amsterdam: Elsevier.
   Google Scholar
• van Rossum, G. et al. (2011), “Python Programming Language,” available at https://www.python.org (last accessed: November 28, 2022).
   Google Scholar
• Vannitsem, S., Bremnes, J. B., Demaeyer, J., Evans, G. R., Flowerdew, J., Hemri, S., Lerch, S., Roberts, N., Theis, S., Atencia, A., Bouallègue, Z. B., Bhend, J., Dabernig, M., Cruz, L. D., Hieta, L., Mestre, O., Moret, L., Plenković, I. O., Schmeits, M., Taillardat, M., den Bergh, J. V., Schaeybroeck, B. V., Whan, K., and Ylhaisi, J. (2021), “Statistical Postprocessing for Weather Forecasts: Review, Challenges, and Avenues in a Big Data World,” Bulletin of the American Meteorological Society, 102, E681–E699. DOI: 10.1175/BAMS-D-19-0308.1.
   Web of Science ®Google Scholar
• Vitart, F., Robertson, A., Spring, A., Pinault, F., Roškar, R., Cao, W., Bech, S., Bienkowski, A., Caltabiano, N., Coning, E. D., Denis, B., Dirkson, A., Dramsch, J., Dueben, P., Gierschendorf, J., Kim, H. S., Nowak, K., Landry, D., Lledó, L., Palma, L., Rasp, S., and Zhou, S. (2022), “Outcomes of the WMO Prize Challenge to Improve Sub-Seasonal to Seasonal Predictions Using Artificial Intelligence,” Bulletin of the American Meteorological Society, 103, E2878–E2886. DOI: 10.1175/BAMS-D-22-0046.1.
   Web of Science ®Google Scholar
• Wang, X., Hyndman, R. J., Li, F., and Kang, Y. (2023), “Forecast Combinations: An over 50-year Review,” International Journal of Forecasting. DOI: 10.1016/j.ijforecast.2022.11.005.
   PubMed Web of Science ®Google Scholar
• White, C. J., Domeisen, D. I., Acharya, N., Adefisan, E. A., Anderson, M. L., Aura, S., Balogun, A. A., Bertram, D., Bluhm, S., Brayshaw, D. J., et al. (2022), “Advances in the Application and Utility of Subseasonal-to-Seasonal Predictions,” Bulletin of the American Meteorological Society, 103, E1448–E1472. DOI: 10.1175/BAMS-D-20-0224.1.
   Web of Science ®Google Scholar
• Witkowski, J., Freeman, R., Vaughan, J. W., Pennock, D. M., and Krause, A. (2022), “Incentive-Compatible Forecasting Competitions,” Management Science, 69, 1354–1374. DOI: 10.1287/mnsc.2022.4410.
   Web of Science ®Google Scholar
• Zängl, G., Reinert, D., Rípodas, P., and Baldauf, M. (2015), “The ICON (ICOsahedral Non-hydrostatic) Modelling Framework of DWD and MPI-M: Description of the Non-hydrostatic Dynamical Core,” Quarterly Journal of the Royal Meteorological Society, 141, 563–579. DOI: 10.1002/qj.2378.
   Web of Science ®Google Scholar
• Zischke, R., Martin, G. M., Frazier, D. T., and Poskitt, D. S. (2022), “The Impact of Sampling Variability on Estimated Combinations of Distributional Forecasts,” Preprint, arXiv:2206.02376.
   Google Scholar