Research map / Detection and segmentation / CNN features
Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vision Mamba (Vim), a pure state-space-model vision backbone that uses bidirectional Mamba blocks with position embeddings to process image patch sequences without attention.
Problem
Mamba-style selective SSMs have not been adapted into a generic, attention-free vision backbone; they are unidirectional and lack positional awareness, which hurts visual representation learning and dense prediction on high-resolution images.
Approach
Process flattened image patch tokens with a Mamba block run in both forward and backward directions (an extra backward SSM plus backward Conv1d per block), with position embeddings and a middle class token, giving a pure-SSM encoder with linear memory and sub-quadratic compute in sequence length.
Key results
- Vim-Ti reaches 76.1% ImageNet top-1 vs 72.2% for DeiT-Ti; Vim-S reaches 80.3% vs 79.8% for DeiT-S, and Vim-B 81.9% vs 81.8% for DeiT-B.
- At 1248x1248 resolution (6084 tokens), Vim is 2.8x faster than DeiT and saves 86.8% GPU memory for batch feature extraction.
- On ADE20K with UperNet, Vim-S gets 44.9 mIoU vs 44.0 for DeiT-S; on COCO with Cascade Mask R-CNN, Vim-Ti gets 45.7 box AP vs 44.4 for DeiT-Ti.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Vim-Ti instead of DeiT-Ti | ImageNet classification · ImageNet-1K val | top-1 accuracy: 72.2 -> 76.1 | Helped |
| Fine-tune Vim models with the long-sequence setting | Vim-Ti, Vim-S, and Vim-B · ImageNet-1K val | top-1 accuracy: 76.1 -> 78.3; 80.3 -> 81.4; 81.9 -> 83.2 | Helped |
| Use Vim-S instead of DeiT-S | ImageNet classification · ImageNet-1K val | top-1 accuracy: 79.8 -> 80.3 | Helped |
| Use Vim-B instead of DeiT-B | ImageNet classification · ImageNet-1K val | top-1 accuracy: 81.8 -> 81.9 | Helped |
| Use Vim backbones instead of DeiT backbones | UperNet · ADE20K val | mIoU: 39.2 -> 41.0; 44.0 -> 44.9 | Helped |
| Use Vim-Ti instead of DeiT-Ti | Cascade Mask R-CNN · COCO val | box AP: 44.4 -> 45.7 | Helped |
| Use Vim-Ti instead of DeiT-Ti | Cascade Mask R-CNN · COCO val | mask AP: 38.1 -> 39.2 | Helped |
| Add a backward SSM and backward Conv1d (ablation) | Vim bidirectional block · ImageNet-1K classification ablation | top-1 accuracy: 73.2 -> 73.9 | Helped |
| Add a backward SSM and backward Conv1d (ablation) | Vim bidirectional block · ADE20K segmentation ablation | mIoU: 32.3 -> 35.9 | Helped |
| Use bidirectional layers (ablation) | Vim · ImageNet-1K classification ablation | top-1 accuracy: 73.2 -> 70.9 | Hurt |
| Use bidirectional layers (ablation) | Vim · ADE20K segmentation ablation | mIoU: 32.3 -> 33.6 | Helped |
| Place the class token in the middle rather than at the head (ablation) | Vim classification head · ImageNet-1K classification ablation | top-1 accuracy: 75.2 -> 76.1 | Helped |
Surprising findings
- A unidirectional Mamba block gives higher ImageNet accuracy (73.2) than the simple bidirectional-block pairing (70.9), though the latter helps segmentation.
- Using a middle class token (76.1) outperforms the standard head class token (75.2) used by DeiT.
Limitations
- Classification gains over DeiT are small at Small and Base sizes (0.5 and 0.1 points).
- Pretraining is only supervised on ImageNet-1K; self-supervised or multimodal pretraining is left to future work.
- Efficiency and dense-task results are reported mainly for the Tiny variant.
Experimental setup
Datasets: ImageNet-1K COCO 2017 ADE20K
Benchmarks: ImageNet-1K COCO 2017 ADE20K
Metrics: top-1 accuracy box AP mask AP mIoU FPS GPU memory
Baselines: DeiT-Ti DeiT-S DeiT-B ResNet-50 ResNet-101 S4ND-ViT-B Swin-T Swin-S Swin-B UperNet Cascade Mask R-CNN
Builds on: Mamba S4 ViT DeiT ViTDet
Closest papers by meaning
- Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model (NeurIPS 2024 · similarity 0.89)
- MambaOut: Do We Really Need Mamba for Vision? (CVPR 2025 · similarity 0.89)
- Mamba-Adaptor: State Space Model Adaptor for Visual Recognition (CVPR 2025 · similarity 0.88)
- EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba (AAAI 2025 · similarity 0.88)
- VMamba: Visual State Space Model (NeurIPS 2024 · similarity 0.88)
- GroupMamba: Efficient Group-Based Visual State Space Model (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.