Auto-WEKA
Chris Thornton, Frank Hutter, Holger H. Hoos et al. · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · 2013
Many different machine learning algorithms exist; taking into account each algorithm's hyperparameters, there is a staggeringly large number of possible alternatives overall. We consider the problem of simultaneously selecting a learning algorithm and setting its hyperparameters, going beyond previous work that attacks these issues separately. We show that this problem can be addressed by a fully automated approach, leveraging recent innovations in Bayesian optimization. Specifically, we consider a wide range of feature selection techniques (combining 3 search and 8 evaluator methods) and all classification approaches implemented in WEKA's standard distribution, spanning 2 ensemble methods, 10 meta-methods, 27 base classifiers, and hyperparameter settings for each classifier. On each of 21 popular datasets from the UCI repository, the KDD Cup 09, variants of the MNIST dataset and CIFAR-10, we show classification performance often much better than using standard selection and hyperparameter optimization methods. We hope that our approach will help non-expert users to more effectively identify machine learning algorithms and hyperparameter settings appropriate to their applications, and hence to achieve improved performance.
Cited by 1,352 works. doi:10.1145/2487575.2487629
Authors
References
- Learning Multiple Layers of Features from Tiny Images
- UCI Machine Learning Repository
- The WEKA data mining software
- Practical Bayesian Optimization of Machine Learning Algorithms
- A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
- Random search for hyper-parameter optimization
- Department of Computer Science
- Efficient Global Optimization of Expensive Black-Box Functions
- Model Selection and Akaike's Information Criterion (AIC): The General Theory and its Analytical Extensions
- Algorithms for Hyper-Parameter Optimization