Scaling distributed machine learning with the parameter server
Mu Li, David G. Andersen, Jun Woo Park et al. · Operating Systems Design and Implementation · 2014
We propose a parameter server framework for distributed machine learning problems. Both data and workloads are distributed over worker nodes, while the server nodes maintain globally shared parameters, represented as dense or sparse vectors and matrices. The framework manages asynchronous data communication between nodes, and supports flexible consistency models, elastic scalability, and continuous fault tolerance.To demonstrate the scalability of the proposed framework, we show experimental results on petabytes of real data with billions of examples and parameters on problems ranging from Sparse Logistic Regression to Latent Dirichlet Allocation and Distributed Sketching.
Cited by 1,021 works. doi:10.5555/2685048.2685095
Authors
- Mu Li
- David G. Andersen
- Jun Woo Park
- Alexander Johannes Smola
- Amr AbdelFatah Ahmed
- Vanja Josifovski
- James E. Long
- Eugene J. Shekita
- Bor-Yiing Su
References
- The Nature of Statistical Learning Theory
- MapReduce
- Latent dirichlet allocation
- The Elements of Statistical Learning
- Large Scale Distributed Deep Networks
- Chord
- Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems
- Pattern Recognition and Machine Learning (Information Science and Statistics)
- Finding scientific topics
- Machine Learning : A Probabilistic Perspective