Hyperparameter Optimization
Matthias Feurer, Frank Hutter · The Springer series on challenges in machine learning · 2019
Recent interest in complex and computationally expensive machine learning models with many hyperparameters, such as automated machine learning (AutoML) frameworks and deep neural networks, has resulted in a resurgence of research on hyperparameter optimization (HPO). In this chapter, we give an overview of the most prominent approaches for HPO. We first discuss blackbox function optimization methods based on model-free methods and Bayesian optimization. Since the high computational demand of many modern machine learning applications renders pure blackbox optimization extremely costly, we next focus on modern multi-fidelity methods that use (much) cheaper variants of the blackbox function to approximately assess the quality of hyperparameter settings. Lastly, we point to open problems and future research directions.
Cited by 1,256 works. doi:10.1007/978-3-030-05318-5_1
Authors
References
- ImageNet Large Scale Visual Recognition Challenge
- Stacked generalization
- Machine Learning, Neural and Statistical Classification
- Lecture Notes in Artificial Intelligence
- Efficient Global Optimization of Expensive Black-Box Functions
- Proceedings of the 25th international conference on Machine learning - ICML '08
- Auto-WEKA
- Comparison between genetic algorithms and particle swarm optimization
- Sequential Model-Based Optimization for General Algorithm Configuration
- Taking the Human Out of the Loop: A Review of Bayesian Optimization