Self-Supervised Learning for Contextualized Extractive Summarization
Hong Wang, Xin Eric Wang, Wenhan Xiong et al. · Annual Meeting of the Association for Computational Linguistics (ACL) · 2019
Existing models for extractive summarization are usually trained from scratch with a crossentropy loss, which does not explicitly capture the global context at the document level.In this paper, we aim to improve this task by introducing three auxiliary pre-training tasks that learn to capture the document-level context in a self-supervised fashion.Experiments on the widely-used CNN/DM dataset validate the effectiveness of the proposed auxiliary tasks.Furthermore, we show that after pretraining, a clean model with simple building blocks is able to outperform previous state-ofthe-art that are carefully designed.1
Cited by 61 works. doi:10.18653/v1/p19-1214 Open-access PDF
Authors
References
- Long Short-Term Memory
- Exploiting Generative AI to Scale up Intelligent Tutoring Systems
- AI-Assisted Pipeline for Dynamic Generation of Trustworthy Health Supplement Content at Scale
- Glove: Global Vectors for Word Representation
- Attention Is All You Need
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- A unified architecture for natural language processing
- Deep Contextualized Word Representations
- ROUGE: A Package for Automatic Evaluation of Summaries
- Self-taught learning