Random search for hyper-parameter optimization
James Bergstra, Yoshua Bengio, Léon Bottou · 2012
Grid search and manual search are the most widely used strategies for hyper-parameter optimization. This paper shows empirically and theoretically that randomly chosen trials are more efficient for hyper-parameter optimization than trials on a grid. Empirical evidence comes from a comparison with a large previous study that used grid search and manual search to configure neural networks and deep belief networks. Compared with neural networks configured by a pure grid search, we find that random search over the same domain is able to find models that are as good or better within a small fraction of the computation time. Granting random search the same computational budget, random search finds better models by effectively searching a larger, less promising configuration space. Compared with deep belief networks configured by a thoughtful combination of manual search and grid search, purely random search over the same 32-dimensional configuration space found statistically equal performance on four of seven data sets, and superior performance on one of seven. A Gaussian process analysis of the function from hyper-parameters to validation set performance reveals that for most data sets only a few of the hyper-parameters really matter, but that different hyper-parameters are important on different data sets. This phenomenon makes
Cited by 7,564 works.
Authors
References
- Gradient-based learning applied to document recognition
- Optimization by Simulated Annealing
- LIBSVM
- A Fast Learning Algorithm for Deep Belief Nets
- A Simplex Method for Function Minimization
- Neural networks for pattern recognition
- Understanding the difficulty of training deep feedforward neural networks
- Learning Deep Architectures for AI
- Extracting and composing robust features with denoising autoencoders
- Efficient BackProp