Research map / Vision-language and multimodal / Human and embodied modeling
VMamba: Visual State Space Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts the Mamba selective state-space model into VMamba, a linear-complexity vision backbone whose 2D Selective Scan (SS2D) traverses images along four scan routes, and reports ImageNet, COCO, and ADE20K results.
Problem
Vision backbones need global receptive fields with efficient scaling to high-resolution inputs, but self-attention has quadratic cost, and Mamba's 1D selective scan does not directly handle 2D image data.
Approach
SS2D unfolds image patches along four scanning paths (cross-scan), runs a separate S6 selective scan on each sequence, and merges the outputs (cross-merge), giving each patch a global receptive field at linear cost; VSS blocks drop the multiplicative branch and add an FFN, and several implementation and architecture changes speed up inference.
Key results
- VMamba-B reaches 83.9% top-1 on ImageNet-1K, surpassing Swin-B by 0.4% with throughput of 646 vs 458 img/s.
- With the Mask R-CNN 1x schedule, VMamba-T/S/B reach 47.3/48.7/49.2 box AP on COCO, outperforming Swin by 4.6/3.9/2.3.
- On ADE20K with single-scale testing, VMamba-T/S/B reach 47.9/50.6/51.0 mIoU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VMamba-T instead of Swin-T | ImageNet-1K classification · ImageNet-1K | Top-1 accuracy: 81.3 -> 82.6 | Helped |
| Use VMamba-B instead of Swin-B | ImageNet-1K classification · ImageNet-1K | Throughput: 646 vs. 458 | Helped |
| Use VMamba-T instead of ConvNeXt-T | COCO Mask R-CNN 1× · COCO | Box AP: 44.2 -> 47.3 | Helped |
| Use VMamba-T instead of ConvNeXt-T | COCO Mask R-CNN 1× · COCO | Mask AP: 40.1 -> 42.7 | Helped |
| Use VMamba-T instead of Swin-T | ADE20K semantic segmentation · ADE20K single-scale | mIoU: +3.4% | Helped |
| Use Triton for Cross-Scan and Cross-Merge (ablation) | VMamba-T · ImageNet-1K | Throughput: +41 img/s | Helped |
| Adjust CUDA selective scan for float16 input and float32 output (ablation) | VMamba-T · ImageNet-1K | Throughput: -3 img/s | Hurt |
| Replace einsum with linear transformation and adopt (B, C, H, W) layout (ablation) | VMamba-T · ImageNet-1K | Throughput: +174 img/s | Helped |
| Introduce MLP, discard DWConv, and reduce layers to [2,2,2,2] (ablation) | VMamba-T · ImageNet-1K | Top-1 accuracy: -0.6% | Hurt |
| Reduce ssm-ratio from 2.0 to 1.0, raise layers to [2,2,5,2], and discard the multiplicative branch (ablation) | VMamba-T · ImageNet-1K | Top-1 accuracy: +0.6% | Helped |
| Introduce DWConv and reduce d_state from 16.0 to 1.0 while raising ssm-ratio to 2.0 (ablation) | VMamba-T · ImageNet-1K | Top-1 accuracy: +0.3% | Helped |
| Reduce ssm-ratio to 1.0 and change layers from [2,2,5,2] to [2,2,8,2] (ablation) | VMamba-T · ImageNet-1K | Top-1 accuracy: +0.1% | Helped |
Surprising findings
- Removing the multiplicative branch and depth-wise convolution, and changing layer counts, reduced FLOPs and sped up inference, yet kept or slightly raised accuracy in some steps.
Limitations
- Pre-training on larger datasets such as ImageNet-21K was not explored, so compatibility of pre-training methods with SSM backbones is unknown.
- Large-scale variants and fine-grained hyperparameter search were not explored due to limited compute.
- Generalization of SS2D to other tasks and modalities remains unexplored.
Experimental setup
Datasets: ImageNet-1K MSCOCO 2017 ADE20K
Benchmarks: ImageNet-1K MSCOCO 2017 ADE20K
Metrics: top-1 accuracy box AP mask AP mIoU throughput FLOPs
Baselines: Swin Transformer ConvNeXt DeiT HiViT XCiT S4ND Vim NAT
Builds on: Mamba S4 Swin Transformer S4ND
Closest papers by meaning
- Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model (NeurIPS 2024 · similarity 0.92)
- EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba (AAAI 2025 · similarity 0.91)
- QuadMamba: Learning Quadtree-based Selective Scan for Visual State Space Model (NeurIPS 2024 · similarity 0.90)
- Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion (ICLR 2025 · similarity 0.90)
- Vision Mamba Mender (NeurIPS 2024 · similarity 0.89)
- GroupMamba: Efficient Group-Based Visual State Space Model (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.