Improving Protein Function Annotation via Unsupervised Pre-training: Robustness, Efficiency, and Insights
David Dohan, Andreea Gane, Maxwell L. Bileschi et al. · 2021
Recent work demonstrated a large ensemble of convolutional neural networks (CNNs) outperforms industry-standard approaches at annotating protein sequences that are far from the training data. These results highlight the potential of deep learning to significantly advance protein sequence annotation, but this particular system is not a practical tool for many biologists because of the computational burden of making predictions using a large ensemble. In this work, we fine-tune a transformer model that is pre-trained on millions of unlabeled natural protein sequences in order to reduce the system's compute burden at prediction time and improve accuracy. By switching from a CNN to the pre-trained transformer, we lift performance from 73.6% to 90.5% using a single model on a challenging clustering-based train-test split, where the ensemble of 59 CNNs achieved 89.0%. Through extensive stratified analysis of model performance, we provide evidence that the new model's predictions are trustworthy, even in cases known to be challenging for prior methods. Finally, we provide a case study of the biological insight enabled by this approach.
Cited by 10 works. doi:10.1145/3447548.3467163 Open-access PDF
Authors
References
- Basic Local Alignment Search Tool
- Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
- Database resources of the National Center for Biotechnology Information: update
- BLAT —The BLAST -Like Alignment Tool
- The SWISS-PROT protein sequence data bank and its supplement TrEMBL in 1998
- The SWISS-PROT protein sequence data bank and its supplement TrEMBL
- Search and clustering orders of magnitude faster than BLAST
- The SWISS-PROT protein sequence database and its supplement TrEMBL in 2000
- Universal Language Model Fine-tuning for Text Classification
- Fast and sensitive protein alignment using DIAMOND