Parallelized Stochastic Gradient Descent
Martin Zinkevich, Markus Weimer, Lihong Li et al. · 2010
With the increase in available data parallel machine learning has become an increasingly pressing problem. In this paper we present the first parallel stochastic gradient descent algorithm including a detailed analysis and experimental evidence. Unlike prior work on parallel optimization algorithms [5, 7] our variant comes with parallel acceleration guarantees and it poses no overly tight latency constraints, which might only be available in the multicore setting. Our analysis introduces a novel proof technique — contractive mappings to quantify the speed of convergence of parameter distributions to their asymptotic limits. As a side effect this answers the question of how quickly stochastic gradient descent algorithms reach the asymptotically normal regime [1, 8]. 1
Cited by 989 works.
Authors
References
- Map-Reduce for Machine Learning on Multicore
- A Theory of Adaptive Pattern Classifiers
- The Tradeoffs of Large-Scale Learning
- Slow Learners are Fast
- Slow Learners are Fast
- Network information criterion-determining the number of hidden units for an artificial neural network model
- Efficient Large-Scale Distributed Training of Conditional Maximum Entropy Models
- Bundle Methods for Regularized Risk Minimization
- Distance-Based Classification with Lipschitz Functions