Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images
Ming Y. Lu, Bowen Chen, Andrew Zhang et al. · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · 2023
Contrastive visual language pretraining has emerged as a powerful method for either training new language-aware image encoders or augmenting existing pretrained models with zero-shot visual recognition capabilities. However, existing works typically train on large datasets of imagetext pairs and have been designed to perform downstream tasks involving only small to medium sized-images, neither of which are applicable to the emerging field of computational pathology where there are limited publicly available paired image-text datasets and each image can span up to 100,000 × 100,000 pixels. In this paper we present MI-Zero, a simple and intuitive framework for unleashing the zero-shot transfer capabilities of contrastively aligned image and text models on gigapixel histopathology whole slide images, enabling multiple downstream diagnostic tasks to be carried out by pretrained encoders without requiring any additional labels. MI-Zero reformulates zero-shot transfer under the framework of multiple instance learning to overcome the computational challenge of inference on extremely large images. We used over 550k pathology reports and other available in-domain text corpora to pretrain our text encoder. By effectively leveraging strong pretrained encoders, our best model pretrained on over 33k histopathology image-caption pairs achieves an average median zero-shot accuracy of 70.2% across three different real-world cancer subtyping tasks. Our code is available at: https://github.com/mahmoodlab/MI-Zero.
Cited by 115 works. doi:10.1109/cvpr52729.2023.01893 Open-access PDF
Authors
- Ming Y. Lu
- Bowen Chen
- Andrew Zhang
- Drew F. K. Williamson
- Richard J. Chen
- Long P. Le
- Yung-Sung Chuang
- Faisal Mahmood
References
- Momentum Contrast for Unsupervised Visual Representation Learning
- High-performance medicine: the convergence of human and artificial intelligence
- MIMIC-III, a freely accessible critical care database
- A Simple Framework for Contrastive Learning of Visual Representations
- Representation Learning with Contrastive Predictive Coding
- Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
- Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer
- Clinical-grade computational pathology using weakly supervised deep learning on whole slide images
- Publicly Available Clinical
- Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing