Contrastive estimation
Noah A. Smith, Jason M. Eisner · Annual Meeting of the Association for Computational Linguistics (ACL) · 2005
Conditional random fields (Lafferty et al., 2001) are quite effective at sequence labeling tasks like shallow parsing (Sha and Pereira, 2003) and named-entity extraction (McCallum and Li, 2003). CRFs are log-linear, allowing the incorporation of arbitrary features into the model. To train on unlabeled data, we require unsupervised estimation methods for log-linear models; few exist. We describe a novel approach, contrastive estimation. We show that the new technique can be intuitively understood as exploiting implicit negative evidence and is computationally efficient. Applied to a sequence labeling problem---POS tagging given a tagging dictionary and unlabeled text---contrastive estimation outperforms EM (with the same feature set), is more robust to degradations of the dictionary, and can largely recover by modeling additional features.
Cited by 305 works. doi:10.3115/1219840.1219884 Open-access PDF
Authors
References
- Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
- Training Products of Experts by Minimizing Contrastive Divergence
- Maximum likelihood estimation from incomplete data via the EM algorithm
- On the limited memory BFGS method for large scale optimization
- Statistical Language Learning
- Unsupervised word sense disambiguation rivaling supervised methods
- Shallow parsing with conditional random fields
- On the algorithmic implementation of multiclass kernel-based vector machines
- Discriminative Reranking for Natural Language Parsing
- Discriminative learning for minimum error classification (pattern recognition)