A Neural Probabilistic Language Model
Yoshua Bengio, Réjean Ducharme, Pascal Vincent et al. · 2000
A goal of statistical language modeling is to learn the joint probability function of sequences of words. This is intrinsically difficult because of the curse of dimensionality: we propose to fight it with its own weapons. In the proposed approach one learns simultaneously (1) a distributed representation for each word (i.e. a similarity between words) along with (2) the probability function for word sequences, expressed with these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar to words forming an already seen sentence. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach very significantly improves on a state-of-the-art trigram model. 1 Introduction A fundamental problem that makes language modeling and other learning problems difficult is the curse of dimensionality. It is par...
Cited by 1,125 works.
Authors
- Yoshua Bengio
- Réjean Ducharme
- Pascal Vincent
- Christian Jauvin
- Jaz K
- Thomas Hofmann
- Tomaso Poggio
- John S. Shawe-Taylor
References
- Indexing by latent semantic analysis
- WordNet: An electronic lexical database . Ed. by Christiane Fellbaum. Cambridge, MA: MIT Press, 1998. Pp. xxii, 423.
- Training Products of Experts by Minimizing Contrastive Divergence
- Finding Structure in Time
- Efficient BackProp
- Class-based n -gram models of natural language
- A maximum entropy approach to natural language processing
- Interpolated estimation of Markov source parameters from sparse data
- Estimation of probabilities from sparse data for the language model component of a speech recognizer
- Distributional clustering of English words