A Generalized Framework for Population Based Training
Ang Li, Ola Spyra, Sagi Perel et al. · ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) · 2019
Population Based Training (PBT) is a recent approach that jointly optimizes neural network weights and hyperparameters which periodically copies weights of the best performers and mutates hyperparameters during training. Previous PBT implementations have been synchronized glass-box systems. We propose a general, black-box PBT framework that distributes many asynchronous "trials" (a small number of training steps with warm-starting) across a cluster, coordinated by the PBT controller. The black-box design does not make assumptions on model architectures, loss functions or training procedures. Our system supports dynamic hyperparameter schedules to optimize both differentiable and non-differentiable metrics. We apply our system to train a state-of-the-art WaveNet generative model for human voice synthesis. We show that our PBT system achieves better accuracy and faster convergence compared to existing methods, given the same computational resource.
Cited by 55 works. doi:10.1145/3292500.3330649 Open-access PDF
Authors
References
- Exploiting Generative AI to Scale up Intelligent Tutoring Systems
- Attention Is All You Need
- Practical Bayesian Optimization of Machine Learning Algorithms
- TensorFlow: A system for large-scale machine learning
- Random search for hyper-parameter optimization
- WaveNet: A Generative Model for Raw Audio
- Completely Derandomized Self-Adaptation in Evolution Strategies
- Librispeech: An ASR corpus based on public domain audio books
- Algorithms for Hyper-Parameter Optimization
- Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting