Boosted Classification Trees and Class Probability/Quantile Estimation
David Mease, Abraham J. Wyner, Andreas Buja · ScholarlyCommons (University of Pennsylvania) · 2007
The standard by which binary classifiers are usually judged, misclassification error, assumes equal costs of misclassifying the two classes or, equivalently, classifying at the 1/2 quantile of the conditional class probability function P[y = 1|x]. Boosted classification trees are known to perform quite well for such problems. In this article we consider the use of standard, off-the-shelf boosting for two more general problems: 1) classification with unequal costs or, equivalently, classification at quantiles other than 1/2, and 2) estimation of the conditional class probability function P[y = 1|x]. We first examine whether the latter problem, estimation of P[y = 1|x], can be solved with Logit-Boost, and with AdaBoost when combined with a natural link function. The answer is negative: both approaches are often ineffective because they overfit P[y = 1|x] even though they perform well as classifiers. A major negative point of the present article is the disconnect between class probability estimation and classification. Next we consider the practice of over/under-sampling of the two classes. We present an algorithm that uses AdaBoost in conjunction with Over/Under-Sampling and Jittering of the data (“JOUS-Boost”). This algorithm is simple, yet successful, and it preserves the advantage of relative protection against overfitting, but for arbitrary misclassification costs and, equivalently, arbitrary quantile boundaries. We then use collections of classifiers obtained from a grid of quantiles to form estimators of class probabilities. The estimates of the class probabilities compare favorably to those obtained by a variety of methods across both simulated and real data sets.
Cited by 243 works. doi:10.5555/1248659.1248675 Open-access PDF
Authors
References
- The Elements of Statistical Learning
- Experiments with a new boosting algorithm
- SMOTE: Synthetic Minority Over-sampling Technique
- Additive logistic regression: a statistical view of boosting (With discussion and a rejoinder by the authors)
- The foundations of cost-sensitive learning
- Logistic Regression, AdaBoost and Bregman Distances
- Elicitation of Personal Probabilities and Expectations
- Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers
- SMOTEBoost: Improving Prediction of the Minority Class in Boosting
- Toward scalable learning with non-uniform class and cost distributions: a case study in credit card fraud detection