Back to overview

Calibration tests for count data

Type of publication Peer-reviewed
Publikationsform Original article (peer-reviewed)
Publication date 2014
Author Wei Wei, Held Leonhard,
Project Statistical methods for spatio-temporal modelling and prediction of infectious diseases
Show all

Original article (peer-reviewed)

Journal TEST
Volume (Issue) 23(4)
Page(s) 787 - 805
Title of proceedings TEST
DOI 10.1007/s11749-014-0380-8


Calibration, the statistical consistency of forecast distributions and observations, is a central requirement for probabilistic predictions. Calibration of continuous forecasts has been widely discussed, and significance tests are commonly used to detect whether a prediction model is miscalibrated. However, calibration tests for discrete forecasts are rare, especially for distributions with unlimited support. In this paper, we propose two types of calibration tests for count data: tests based on conditional exceedance probabilities and tests based on proper scoring rules. For the latter, three scoring rules are considered: the ranked probability score, the logarithmic score and the Dawid-Sebastiani score. Simulation studies show that all the different tests have good control of the type I error rate and sufficient power under miscalibration. As an illustration, we apply the methodology to weekly data on meningoccocal disease incidence in Germany, 2001--2006. The results show that the test approach is powerful in detecting miscalibrated forecasts.