Parallelizing Word2Vec in Shared and Distributed Memory
Shihao Ji, Nadathur Satish, Sheng Li et al. · IEEE Transactions on Parallel and Distributed Systems · 2019
Word2vec is a widely used algorithm for extracting low-dimensional vector representations of words. State-of-the-art algorithms including those by Mikolov et al. [1] , [2] have been parallelized for multi-core CPU architectures, but are based on vector-vector operations with “Hogwild” updates that are memory-bandwidth intensive and do not efficiently use computational resources. In this paper, we propose “HogBatch” by improving reuse of various data structures in the algorithm through the use of minibatching and negative sample sharing, hence allowing us to express the problem using matrix multiply operations. We also explore different techniques to distribute word2vec computation across nodes in a computer cluster, and demonstrate good strong scalability up to 32 nodes. The new algorithm is particularly suitable for modern multi-core/many-core architectures, especially Intel's latest Knights Landing processors, and allows us to scale up the computation near linearly across cores and nodes, and process hundreds of millions of words per second, which is the fastest word2vec implementation to the best of our knowledge. We released the source code for reproducible research and general usage.
Cited by 74 works. doi:10.1109/tpds.2019.2904058 Open-access PDF
Authors
References
- A Stochastic Approximation Method
- Efficient Estimation of Word Representations in Vector Space
- Sequence to Sequence Learning with Neural Networks
- Distributed Representations of Words and Phrases and their Compositionality
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
- Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
- Foundations of statistical natural language processing
- A unified architecture for natural language processing
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Effective Approaches to Attention-based Neural Machine Translation