Asynchronous Stochastic Gradient Descent with Delay Compensation for Distributed Deep Learning.
Shuxin Zheng, Qi Meng, Taifeng Wang et al. · arXiv (Cornell University) · 2016
With the fast development of deep learning, it has become common to learn big neural networks using massive training data. Asynchronous Stochastic Gradient Descent (ASGD) is widely adopted to fulfill this task for its efficiency, which is, however, known to suffer from the problem of delayed gradients. That is, when a local worker adds its gradient to the global model, the global model may have been updated by other workers and this gradient becomes delayed. We propose a novel technology to compensate this delay, so as to make the optimization behavior of ASGD closer to that of sequential SGD. This is achieved by leveraging Taylor expansion of the gradient function and efficient approximation to the Hessian matrix of the loss function. We call the new algorithm Delay Compensated ASGD (DC-ASGD). We evaluated the proposed algorithm on CIFAR-10 and ImageNet datasets, and the experimental results demonstrate that DC-ASGD outperforms both synchronous SGD and asynchronous SGD, and nearly approaches the performance of sequential SGD.
Cited by 25 works.
Authors
References
- Deep Residual Learning for Image Recognition
- ImageNet Large Scale Visual Recognition Challenge
- AutoResearch Lite: a reproducible paper-inspired experiment runner (Adam mechanism reproduction and convergence-speed comparison)
- Learning Multiple Layers of Features from Tiny Images
- Large Scale Distributed Deep Networks
- Distributed Representations of Words and Phrases and their Compositionality
- Improving the convergence of back-propagation learning with second-order methods
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- Stochastic Gradient Descent Tricks
- Revisiting Distributed Synchronous SGD