Segmenter: Transformer for Semantic Segmentation
Robin Strudel, Ricardo Alexandrino Garcia, Ivan Laptev A et al. · IEEE/CVF International Conference on Computer Vision (ICCV) · 2021
Image segmentation is often ambiguous at the level of individual image patches and requires contextual information to reach label consensus. In this paper we introduce Segmenter, a transformer model for semantic segmentation. In contrast to convolution-based methods, our approach allows to model global context already at the first layer and throughout the network. We build on the recent Vision Transformer (ViT) and extend it to semantic segmentation. To do so, we rely on the output embeddings corresponding to image patches and obtain class labels from these embed-dings with a point-wise linear decoder or a mask trans-former decoder. We leverage models pre-trained for image classification and show that we can fine-tune them on moderate sized datasets available for semantic segmentation. The linear decoder allows to obtain excellent results already, but the performance can be further improved by a mask transformer generating class masks. We conduct an extensive ablation study to show the impact of the different parameters, in particular the performance is better for large models and small patch sizes. Segmenter attains excellent results for semantic segmentation. It outperforms the state of the art on both ADE20K and Pascal Context datasets and is competitive on Cityscapes.
Cited by 2,051 works. doi:10.1109/iccv48922.2021.00717 Open-access PDF
Authors
References
- ImageNet: A large-scale hierarchical image database
- U-Net: Convolutional Networks for Biomedical Image Segmentation
- Fully convolutional networks for semantic segmentation
- Dropout: a simple way to prevent neural networks from overfitting
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- A Stochastic Approximation Method
- Attention Is All You Need
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
- SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation