ADADELTA: An Adaptive Learning Rate Method
Matthew D. Zeiler · arXiv (Cornell University) · 2012
We present a novel per-dimension learning rate method for gradient descent called ADADELTA. The method dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent. The method requires no manual tuning of a learning rate and appears robust to noisy gradient information, different model architecture choices, various data modalities and selection of hyperparameters. We show promising results compared to other methods on the MNIST digit classification task using a single machine and on a large scale voice dataset in a distributed cluster environment.
Cited by 5,122 works. doi:10.48550/arxiv.1212.5701 Open-access PDF
Authors
References
- Learning representations by back-propagating errors
- A Stochastic Approximation Method
- Large Scale Distributed Deep Networks
- Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition
- Improving the convergence of back-propagation learning with second-order methods
- No More Pesky Learning Rates