Probabilistic Forecasts, Calibration and Sharpness
Tilmann Gneiting, Fadoua Balabdaoui, Adrian E. Raftery · Journal of the Royal Statistical Society Series B (Statistical Methodology) · 2007
Summary Probabilistic forecasts of continuous variables take the form of predictive densities or predictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Calibration refers to the statistical consistency between the distributional forecasts and the observations and is a joint property of the predictions and the events that materialize. Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only. A simple theoretical framework allows us to distinguish between probabilistic calibration, exceedance calibration and marginal calibration. We propose and study tools for checking calibration and sharpness, among them the probability integral transform histogram, marginal calibration plots, the sharpness diagram and proper scoring rules. The diagnostic approach is illustrated by an assessment and ranking of probabilistic forecasts of wind speed at the Stateline wind energy centre in the US Pacific Northwest. In combination with cross-validation or in the time series context, our proposal provides very general, nonparametric alternatives to the use of information criteria for model diagnostics and model selection.
Cited by 1,758 works. doi:10.1111/j.1467-9868.2007.00587.x
Authors
References
- VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
- ON A METHOD OF DETERMINING WHETHER A SAMPLE OF SIZE n SUPPOSED TO HAVE BEEN DRAWN FROM A PARENT POPULATION HAVING A KNOWN PROBABILITY INTEGRAL HAS PROBABLY BEEN DRAWN AT RANDOM
- Comparing Predictive Accuracy
- Remarks on a Multivariate Transformation
- Strictly Proper Scoring Rules, Prediction, and Estimation
- Comparing Predictive Accuracy
- Bayesian Computation and Stochastic Systems
- Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician
- An Overview of Value at Risk
- Evaluating Interval Forecasts