Predicting good probabilities with supervised learning
Alexandru Niculescu-Mizil, Rich Caruana · International Conference on Machine Learning · 2005
We examine the relationship between the predictions made by different learning algorithms and true posterior probabilities. We show that maximum margin methods such as boosted trees and boosted stumps push probability mass away from 0 and 1 yielding a characteristic sigmoid shaped distortion in the predicted probabilities. Models such as Naive Bayes, which make unrealistic independence assumptions, push probabilities toward 0 and 1. Other models such as neural nets and bagged trees do not have these biases and predict well calibrated probabilities. We experiment with two ways of correcting the biased probabilities predicted by some learning methods: Platt Scaling and Isotonic Regression. We qualitatively examine what kinds of distortions these calibration methods are suitable for and quantitatively examine how much data they need to be effective. The empirical results show that after calibration boosted trees, random forests, and SVMs predict the best probabilities.
Cited by 1,766 works. doi:10.1145/1102351.1102430
Authors
References
- UCI Repository of machine learning databases
- Making Large-Scale SVM Learning Practical
- Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
- Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions
- An Empirical Distribution Function for Sampling with Incomplete Information
- Order Restricted Statistical Inference.
- Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers
- Transforming classifier scores into accurate multiclass probability estimates
- Tree Induction for Probability-Based Ranking
- The Comparison and Evaluation of Forecasters