Statistical Comparisons of Classifiers over Multiple Data Sets
Janez Demšar · 2006
While methods for comparing two learning algorithms on a single data set have been scrutinized for quite some time already, the issue of statistical tests for comparisons of more algorithms on multiple data sets, which is even more essential to typical machine learning studies, has been all but ignored. This article reviews the current practice and then theoretically and empirically examines several suitable tests. Based on that, we recommend a set of simple, yet safe and robust non-parametric tests for statistical comparisons of classifiers: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparison of more classifiers over multiple data sets. Results of the latter can also be neatly presented with the newly introduced CD (critical difference) diagrams.
Cited by 11,020 works.
Authors
References
- A Simple Sequentially Rejective Multiple Test Procedure
- UCI Repository of machine learning databases
- Robust Locally Weighted Regression and Smoothing Scatterplots
- A Multiple Comparison Procedure for Comparing Several Treatments with a Control
- A Sharper Bonferroni Procedure for Multiple Tests of Significance
- The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance
- Comparing Individual Means in the Analysis of Variance
- A Stagewise Rejective Multiple Test Procedure Based on a Modified Bonferroni Test
- Bagging, boosting, and C4.S
- Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms