Research map / Architectures and efficiency / Attention and transformers
Your ViT is Secretly an Image Segmentation Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that large, well-pretrained plain ViTs (DINOv2) can replace task-specific adapters, pixel decoders and Transformer decoders for image segmentation, and proposes EoMT, a ViT-only segmenter with mask annealing that drops masked attention at inference.
Problem
ViT-based image segmentation relies on task-specific components (ViT-Adapter, pixel decoder, Transformer decoder with masked attention) that add complexity and latency; it is unclear whether they remain necessary with large models and extensive pre-training.
Approach
Progressively remove task-specific components from ViT-Adapter + Mask2Former, then repurpose the last ViT blocks to jointly process learnable queries and patch tokens (Encoder-only Mask Transformer), with mask annealing that phases out masked attention during training so inference uses plain ViT attention.
Key results
- EoMT with ViT-L reaches 56.0 PQ on COCO val2017 at 128 FPS, versus 54.4 PQ at 32 FPS for ViT-Adapter + Mask2Former with ViT-B.
- Removing masked attention via mask annealing speeds EoMT ViT-L up 2.1x with PQ 56.2 to 56.0; w/o annealing (no masking in training) PQ falls to 53.2.
- With ViT-g, EoMT gets 57.0 PQ vs 57.7 for ViT-Adapter + M2F, a gap of 0.7 PQ; with DINOv2 pretraining the gap to M2F is 1.1 PQ at ViT-L, versus 3.9 and 6.1 for IN21K and IN1K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove ViT-Adapter (ablation) | ViT-Adapter + Mask2Former · COCO val2017 panoptic | PQ: 57.1 -> 56.7 | Hurt |
| remove pixel decoder (ablation) | ViT-Adapter + Mask2Former without ViT-Adapter · COCO val2017 panoptic | PQ: 56.7 -> 56.9 | No clear effect |
| remove multi-scale feature processing (ablation) | Mask2Former without ViT-Adapter and pixel decoder · COCO val2017 panoptic | PQ: 56.9 -> 56.7 | Hurt |
| remove Transformer decoder (ablation) | model without ViT-Adapter, pixel decoder, and multi-scale processing · COCO val2017 panoptic | PQ: 56.7 -> 56.2 | Hurt |
| remove masked attention at inference after mask annealing (ablation) | EoMT · COCO val2017 panoptic | PQ: 56.2 -> 56.0 | No clear effect |
| use DINOv2 pre-training instead of ImageNet-1K pre-training | EoMT · COCO val2017 panoptic | PQ: 44.3 -> 56.0 | Helped |
| scale the backbone from ViT-S to ViT-L | EoMT · COCO val2017 panoptic | PQ: 44.7 -> 56.0 | Helped |
| disable masked attention only at inference after training with masking (ablation) | EoMT · COCO val2017 panoptic | PQ: 56.2 -> 27.4 | Hurt |
| disable masked attention during both training and inference (ablation) | EoMT · COCO val2017 panoptic | PQ: 56.2 -> 53.2 | Hurt |
| use mask annealing to phase out masked attention during training (ablation) | EoMT · COCO val2017 panoptic | PQ: 56.2 -> 56.0 | No clear effect |
| apply ALGM token merging (ablation) | EoMT · Cityscapes val semantic | mIoU: 84.2 -> 84.2 | No clear effect |
Surprising findings
- Removing the ViT-Adapter, pixel decoder and multi-scale features reduces PQ by only 0.4 while nearly doubling FPS, and the full removal of masked attention keeps PQ near 56.0.
- Models trained with masked attention but run without it lose severely (PQ 27.4), showing train-inference mismatch is harmful.
Limitations
- Gap to the complex baseline widens for smaller models and weaker pretraining (e.g., 5.8 PQ for ViT-S, 6.1 mIoU-like drops with IN1K).
- Instance segmentation AP drop is larger than for panoptic and semantic tasks (45.2 vs 47.6 AP for ViT-L).
Experimental setup
Datasets: COCO ADE20K Cityscapes BRAVO
Benchmarks: COCO val2017 ADE20K val Cityscapes val BRAVO
Metrics: PQ mIoU AP FPS GFLOPs
Baselines: ViT-Adapter + Mask2Former Mask2Former kMaX-DeepLab OneFormer MaskDINO UViT YOLOS
Builds on: Mask2Former ViT-Adapter ViTDet DINOv2
Closest papers by meaning
- The Missing Point in Vision Transformers for Universal Image Segmentation (CVPR 2026 · similarity 0.91)
- VidEoMT: Your ViT is Secretly Also a Video Segmentation Model (CVPR 2026 · similarity 0.90)
- Vision Transformer Adapter for Dense Predictions (ICLR 2023 · similarity 0.89)
- Intriguing Properties of Vision Transformers (NeurIPS 2021 · similarity 0.89)
- AiluRus: A Scalable ViT Framework for Dense Prediction (NeurIPS 2023 · similarity 0.89)
- DDViT: Double-Level Fusion Domain Adapter Vision Transformer (Student Abstract) (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.