Research map / Architectures and efficiency / Attention and transformers
Demystify Mamba in Vision: A Linear Attention Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates Mamba's selective SSM as a linear attention variant, ablates six design differences on ImageNet with Swin-based linear attention, and proposes MILA, a linear attention vision model that replaces the forget gate with positional encodings (LePE, CPE, RoPE) and adopts Mamba's block design.
Problem
Mamba performs well in vision while linear attention Transformers usually underperform; the paper asks which design factors explain Mamba's advantage over linear attention.
Approach
Shows selective SSM is a single-head linear attention with input gate, forget gate, shortcut, no normalization, and a modified block; finds forget gate and block design matter most, and that positional encodings can replace the forget gate in parallel linear attention, yielding MILA.
Key results
- MILA-T reaches 83.5 Top-1 on ImageNet-1K with 25M params and 4.2G FLOPs, versus 82.7 for LocalVMamba-T and 82.5 for VMamba-T.
- MILA-B reaches 85.3 Top-1 on ImageNet-1K, outperforming VMamba-B (83.9) by a clear margin.
- In the ablation on Swin-based linear attention, the forget gate raises Top-1 from 77.6 to 78.4 but lowers throughput from 1152 to 743; replacing it with LePE or CPE gives 81.6 or 81.7 at similar throughput.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add input gate (ablation) | Baseline linear attention · ImageNet-1K val | Top-1: 77.6 -> 77.8 | Helped |
| add forget gate (ablation) | Baseline linear attention · ImageNet-1K val | Top-1: 77.6 to 78.4 | Helped |
| add forget gate (ablation) | Baseline linear attention · ImageNet-1K val | Throughput: 1152 to 743 | Hurt |
| replace forget gate with positional encoding (ablation) | Baseline linear attention · ImageNet-1K val | Top-1: 77.6 -> 80.0 (APE); 81.6 (LePE); 81.7 (CPE); 80.0 (RoPE) | Helped |
| add shortcut (ablation) | Baseline linear attention · ImageNet-1K val | Top-1: 77.6 -> 77.8 | Helped |
| remove attention normalization (ablation) | Baseline linear attention · ImageNet-1K val | Top-1: 77.6 to 72.4 | Hurt |
| remove multi-head design (ablation) | Baseline linear attention · ImageNet-1K val | Top-1: 77.6 -> 73.5 | Hurt |
| replace full Transformer block with Mamba block design (ablation) | Baseline linear attention · ImageNet-1K val | Top-1: 77.6 -> 79.4 | Helped |
| replace attention sub-block with Mamba block design (ablation) | Baseline linear attention · ImageNet-1K val | Top-1: 77.6 -> 80.9 | Helped |
| add MESA (ablation) | MILA · ImageNet-1K val | Top-1: 83.3 -> 83.5 (MILA-T); 84.2 -> 84.3 (MILA-S); 85.0 -> 85.3 (MILA-B) | Helped |
| use MILA-T instead of NAT-T | ImageNet classification · ImageNet-1K val | Top-1: 83.2 -> 83.5 | Helped |
| use MILA-B instead of VMamba-B | COCO Mask R-CNN 1x · COCO Mask R-CNN 1x, box | AP: 49.2 -> 50.5 | Helped |
Surprising findings
- Removing attention normalization drops accuracy sharply from 77.6 to 72.4, showing normalization is important for linear attention.
- Positional encodings (LePE, CPE, RoPE) substituted for the forget gate outperform the forget gate itself in accuracy while maintaining higher throughput.
Limitations
- Other small implementation differences between Mamba and linear attention Transformers may exist, so the analysis is not exhaustive.
- Ablations are run by separately adding each design to a Swin-based linear attention baseline rather than jointly.
- Experiments are limited to vision tasks (ImageNet, COCO, ADE20K).
Experimental setup
Datasets: ImageNet-1K COCO ADE20K
Benchmarks: ImageNet-1K COCO ADE20K
Metrics: Top-1 accuracy AP^b AP^m mIoU throughput FLOPs #Params
Baselines: VMamba LocalVMamba MambaOut Vim Mamba2D PlainMamba EfficientVMamba ConvNeXt Swin Transformer Hydra Attention Efficient Attention FLatten Transformer
Builds on: Mamba linear attention Swin Transformer FLatten Transformer VMamba
Closest papers by meaning
- Vision Mamba Mender (NeurIPS 2024 · similarity 0.93)
- MambaOut: Do We Really Need Mamba for Vision? (CVPR 2025 · similarity 0.92)
- Mamba-Adaptor: State Space Model Adaptor for Visual Recognition (CVPR 2025 · similarity 0.91)
- MambaVision: A Hybrid Mamba-Transformer Vision Backbone (CVPR 2025 · similarity 0.90)
- Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model (NeurIPS 2024 · similarity 0.90)
- MobileMamba: Lightweight Multi-Receptive Visual Mamba Network (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.