Sparse Overcomplete Word Vector Representations
Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama et al. · arXiv (Cornell University) · 2015
Current distributed representations of words show little resemblance to theories of lexical semantics. The former are dense and uninterpretable, the latter largely based on familiar, discrete classes (e.g., supersenses) and relations (e.g., synonymy and hypernymy). We propose methods that transform word vectors into sparse (and optionally binary) vectors. The resulting representations are more similar to the interpretable features typically used in NLP, though they are discovered automatically from raw corpora. Because the vectors are highly sparse, they are computationally easy to work with. Most importantly, we find that they outperform the original vectors on benchmark tasks.
Cited by 10 works. doi:10.48550/arxiv.1506.02004 Open-access PDF
Authors
References
- The Measurement of Observer Agreement for Categorical Data
- WordNet
- Efficient Estimation of Word Representations in Vector Space
- Glove: Global Vectors for Word Representation
- Building a Large Annotated Corpus of English: The Penn Treebank
- Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
- Learning the parts of objects by non-negative matrix factorization
- Representation Learning: A Review and New Perspectives
- Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
- Sparse coding with an overcomplete basis set: A strategy employed by V1?