Research map / Architectures and efficiency / Attention and transformers
Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MSVMamba, a vision state space backbone that replaces VMamba's SS2D with a multi-scale 2D scan (full-resolution plus downsampled branches) and adds a ConvFFN channel mixer, achieving competitive ImageNet, COCO and ADE20K results with fewer FLOPs.
Problem
Multi-scan strategies in vision Mamba models improve accuracy but add substantial computational redundancy, and the S6 long-range forgetting problem limits parameter-constrained vision SSMs.
Approach
Apply multi-directional scans at two resolutions: one full-resolution scan preserves fine details, while three scans on a 2x-downsampled map (sharing one S6 block) shorten token distances and cut cost; a ConvFFN is added as channel mixer.
Key results
- MSVMamba-T reaches 82.8% ImageNet top-1 with 33M params and 4.6G FLOPs, +0.6% over VMamba-T with fewer FLOPs.
- MSVMamba-T achieves 46.9 box AP on Mask R-CNN 1x, +4.2 over Swin-T.
- MSVMamba-T obtains 47.6 single-scale mIoU on ADE20K, +2.2 over Swin-T and +0.3 over VMamba-T.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace SS2D with MS2D at roughly equivalent FLOPs (ablation) | VMamba-Nano · ImageNet-1K | Top-1 accuracy: 69.6% to 71.9% | Helped |
| add an SE block after MS2D (ablation) | MSVMamba-Nano · ImageNet-1K | Top-1 accuracy: 71.9 to 72.4; 0.5% | Helped |
| add ConvFFN as a channel mixer (ablation) | MSVMamba-Nano with MS2D + SE · ImageNet-1K | Top-1 accuracy: 72.4 to 74.4; 2.0% | Helped |
| set state space dimension N=1 and add one block (ablation) | MSVMamba-Nano with MS2D + SE + ConvFFN · ImageNet-1K | Top-1 accuracy: 74.4 to 75.1; 0.7% | Helped |
| use four half-resolution scans and no full-resolution scan (ablation) | MS2D · ImageNet-1K | Top-1 accuracy: 71.9 to 63.1 | Hurt |
| use two rather than one full-resolution scan (ablation) | MS2D · ImageNet-1K | Top-1 accuracy: 71.9 to 72.0; 0.1% | Helped |
| use three rather than one full-resolution scan (ablation) | MS2D · ImageNet-1K | Top-1 accuracy: 71.9 to 72.5; 0.6% | Helped |
| use four full-resolution scans instead of the default multi-scale scan (ablation) | MS2D · ImageNet-1K | Top-1 accuracy: 71.9 to 72.3; 0.4%; computational cost by 34% | Mixed |
| use MSVMamba-T instead of VMamba-T | ImageNet-1K classification · ImageNet-1K | Top-1 accuracy: 82.2 to 82.8; 0.6% | Helped |
| use MSVMamba-T instead of VMamba-T | UperNet segmentation · ADE20K single-scale | mIoU: 47.3 to 47.6; +0.3 | Helped |
Surprising findings
- Placing all scans on the downsampled branch drops VMamba-Nano accuracy from 69.6% to 63.1%, showing fine-grained full-resolution scanning is essential.
Limitations
- Scalability to larger models is unexplored; the multi-scale design may give only marginal gains when model size grows.
- Gains are reported mainly for parameter-limited models and the benefit may shrink at larger scales.
Experimental setup
Datasets: ImageNet-1K MSCOCO ADE20K
Benchmarks: ImageNet-1K MSCOCO ADE20K
Metrics: top-1 accuracy box AP mask AP mIoU FLOPs throughput
Baselines: VMamba-T LocalVMamba-T EffVMamba-B ViM-T Swin-T ConvNeXt-T RegNetY DeiT LightViT-T PVT-T
Builds on: VMamba Mamba SS2D EfficientVMamba ViM
Closest papers by meaning
- EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba (AAAI 2025 · similarity 0.92)
- VMamba: Visual State Space Model (NeurIPS 2024 · similarity 0.92)
- GroupMamba: Efficient Group-Based Visual State Space Model (CVPR 2025 · similarity 0.91)
- QuadMamba: Learning Quadtree-based Selective Scan for Visual State Space Model (NeurIPS 2024 · similarity 0.90)
- Vision Mamba Mender (NeurIPS 2024 · similarity 0.90)
- Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.