Research map / Architectures and efficiency / Attention and transformers
Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Spatial-Mamba, a visual SSM that adds a dilated-convolution structure-aware state fusion step over states from a single unidirectional scan, reaching or surpassing multi-scan visual SSMs on ImageNet, COCO and ADE20K.
Problem
Visual state space models flatten images into 1D sequences with multiple scanning orders, which distorts 2D spatial relations and adds computational cost from lengthened scan paths.
Approach
Adds a structure-aware state fusion (SASF) equation that linearly combines hidden states from spatial neighbors (via multi-scale dilated depth-wise convolutions, d=1,3,5) after a unidirectional scan, so each state is directly connected to its 2D neighborhood; the paper also writes Mamba, linear attention and Spatial-Mamba as y = Mu with structured M matrices.
Key results
- Spatial-Mamba-T reaches 83.5% top-1 on ImageNet-1K, vs. VMamba-T 82.6% and LocalVMamba-T 82.7%; Spatial-Mamba-B reaches 85.3%.
- With Mask R-CNN 1x schedule on COCO, Spatial-Mamba-T gets 47.6 box AP and 42.9 mask AP, vs. VMamba-T 47.3 / 42.7.
- On ADE20K with UPerNet, Spatial-Mamba-T achieves 48.6 single-scale mIoU vs. VMamba-T 48.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add structure-aware state fusion (ablation) | Spatial-Mamba-T · ImageNet-1K | Top-1 accuracy: 82.0 -> 82.3 | Helped |
| Increase the SASF neighbor set from 3×3 to 5×5 (ablation) | Spatial-Mamba-T · ImageNet-1K | Top-1 accuracy: 82.3 -> 82.5 | Helped |
| Use dilated SASF neighbors with d=1,3,5 (ablation) | Spatial-Mamba-T · ImageNet-1K | Top-1 accuracy: 82.5 -> 82.7 | Helped |
| Use dilated SASF neighbors with d=1,3,5 (ablation) | Spatial-Mamba-T · ImageNet-1K | Throughput: 1461 -> 1209 | Hurt |
| Add overlapped stem (ablation) | Spatial-Mamba-T with dilated neighbor set · ImageNet-1K | Top-1 accuracy: 82.7 -> 82.9 | Helped |
| Add LPU (ablation) | Spatial-Mamba-T with overlapped stem · ImageNet-1K | Top-1 accuracy: 82.9 -> 83.3 | Helped |
| Add re-parameterization and optimize CUDA kernels (ablation) | Spatial-Mamba-T with LPU · ImageNet-1K | Throughput: 1065 to 1438 | Helped |
| Add MESA (ablation) | Spatial-Mamba-T with re-parameterization · ImageNet-1K | Top-1 accuracy: 83.3 -> 83.5 | Helped |
| Add SASF (ablation) | Vim-Ti · ImageNet-1K | Top-1 accuracy: 71.7 -> 72.2 | Helped |
| Add SASF (ablation) | VMamba-T · ImageNet-1K | Top-1 accuracy: 82.6 -> 82.9 | Helped |
| Replace depth-wise convolution with dynamic convolution (ablation) | Spatial-Mamba-T · ImageNet-1K | Top-1 accuracy: 83.5 -> 83.7 | Helped |
| Replace depth-wise convolution with dynamic convolution (ablation) | Spatial-Mamba-T · ImageNet-1K | Throughput: 1438 to 907 | Hurt |
Surprising findings
- Dynamic convolution as the fusion operator gives only a 0.2% gain while cutting throughput from 1438 to 907.
- Spatial-Mamba-T has lower throughput (1438) than VMamba-T (1686) on ImageNet despite the single-scan design.
Limitations
- Spatial-Mamba-T is slower than VMamba-T in throughput.
- Fusion operator choice trades accuracy against throughput (dynamic conv is slower).
Experimental setup
Datasets: ImageNet-1K COCO 2017 ADE20K
Benchmarks: ImageNet-1K COCO 2017 ADE20K
Metrics: top-1 accuracy throughput box AP mask AP mIoU FLOPs parameters
Baselines: VMamba LocalVMamba Vim Swin Transformer ConvNeXt NAT MambaOut S4ND
Builds on: Mamba VMamba Vim LocalVMamba MESA
Closest papers by meaning
- QuadMamba: Learning Quadtree-based Selective Scan for Visual State Space Model (NeurIPS 2024 · similarity 0.91)
- 2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification (CVPR 2025 · similarity 0.91)
- Vision Mamba Mender (NeurIPS 2024 · similarity 0.90)
- Mamba-Adaptor: State Space Model Adaptor for Visual Recognition (CVPR 2025 · similarity 0.90)
- Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model (NeurIPS 2024 · similarity 0.90)
- VMamba: Visual State Space Model (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.