cdec: A Decoder, Alignment, and Learning Framework for Finite-State and Context-Free Translation Models
Chris Dyer, Adam Lopez, Juri Ganitkevitch et al. · 2010
We present cdec, an open source frame-work for decoding, aligning with, and training a number of statistical machine translation models, including word-based models, phrase-based models, and models based on synchronous context-free gram-mars. Using a single unified internal representation for translation forests, the decoder strictly separates model-specific translation logic from general rescoring, pruning, and inference algorithms. From this unified representation, the decoder can extract not only the 1- or k-best transla-tions, but also alignments to a reference, or the quantities necessary to drive dis-criminative training using gradient-based or gradient-free optimization techniques. Its efficient C++ implementation means that memory use and runtime performance are significantly better than comparable decoders. 1
Cited by 224 works.
Authors
- Chris Dyer
- Adam Lopez
- Juri Ganitkevitch
- Jonathan Weese
- Hendra Setiawan
- Ferhan Türe
- Vladimir Eidelman
- Phil Blunsom
- Philip Resnik
References
- MapReduce
- BLEU
- The mathematics of statistical machine translation: parameter estimation
- Statistical phrase-based translation
- On the limited memory BFGS method for large scale optimization
- SRILM - an extensible language modeling toolkit
- Moses
- A direct adaptive method for faster backpropagation learning: the RPROP algorithm
- Minimum error rate training in statistical machine translation
- Shallow parsing with conditional random fields