Research map / Detection and segmentation / CNN features
MambaVision: A Hybrid Mamba-Transformer Vision Backbone
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MambaVision, a hierarchical hybrid backbone that redesigns the Mamba mixer for vision (regular convs, symmetric non-SSM branch) and places self-attention in the final layers of stages 3-4, achieving SOTA ImageNet-1K accuracy-throughput trade-offs and competitive COCO/ADE20K results.
Problem
Pure Mamba/SSM vision backbones lag ViT and CNN backbones on vision tasks because the autoregressive scan limits global context and spatial modeling; the paper asks how to adapt Mamba for vision and combine it with Transformers.
Approach
Redesign the Mamba mixer for vision by replacing causal convolutions with regular convolutions and adding a symmetric non-SSM branch concatenated with the SSM branch, then build a hierarchical hybrid where CNN blocks handle early high-resolution stages and self-attention is placed in the final layers of the Mamba stages.
Key results
- MambaVision-B reaches 84.2% ImageNet-1K Top-1 vs. 83.9% for VMamba-B and 83.8% for ConvNeXt-B, with higher throughput (3670 img/s).
- MambaVision-L2 reaches 85.3% Top-1 on ImageNet-1K at 224 resolution; ImageNet-21K pretraining raises MambaVision-B to 84.9% and MambaVision-L to 86.1%.
- On ADE20K with UperNet, MambaVision-T/S/B achieve 46.0/48.2/49.1 mIoU, outperforming Swin-T/S/B by +1.5/+0.6/+1.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace causal conv1 with regular convolution (ablation) | MambaVision-T token mixer · ImageNet-1K | Top-1 accuracy: 80.5 -> 80.9 | Helped |
| Add conv2 while retaining the original gating mechanism (ablation) | MambaVision-T token mixer · ImageNet-1K | Top-1 accuracy: 80.9 -> 81.3 | Helped |
| Concatenate outputs from the SSM and non-SSM branches (ablation) | MambaVision-T token mixer · ImageNet-1K | Top-1 accuracy: +1.0% | Helped |
| Concatenate outputs from the SSM and non-SSM branches (ablation) | MambaVision-T token mixer · MS COCO | Box AP: +1.1 | Helped |
| Concatenate outputs from the SSM and non-SSM branches (ablation) | MambaVision-T token mixer · MS COCO | Mask AP: +0.8 | Helped |
| Concatenate outputs from the SSM and non-SSM branches (ablation) | MambaVision-T token mixer · ADE20K | mIoU: +0.9 | Helped |
| Place self-attention blocks in the final N/2 layers rather than use a random integration pattern (ablation) | MambaVision-T · ImageNet-1K | Top-1 accuracy: 81.3% to 82.3% | Helped |
| Alternate self-attention and MambaVision blocks, starting with self-attention (ablation) | MambaVision-T · ImageNet-1K | Top-1 accuracy: -0.1% | Hurt |
| Pre-train MambaVision-B on ImageNet-21K | MambaVision-B · ImageNet-1K, 224 resolution | Top-1 accuracy: 84.2% to 84.9% | Helped |
| Pre-train and fine-tune MambaVision-L | MambaVision-L · ImageNet-1K, 224 resolution | Top-1 accuracy: 85% to 86.1% | Helped |
| Use MambaVision-T instead of Swin-T | Image classification backbone · ADE20K | mIoU: +1.5 | Helped |
| Use MambaVision-S instead of Swin-S | Image classification backbone · ADE20K | mIoU: +0.6 | Helped |
Surprising findings
- Placing self-attention in the last N/2 layers of each stage outperforms random or interleaved placements (82.3% vs 81.3% random on ImageNet-1K top-1 in the MambaVision-T ablation).
Limitations
- Downstream detection and segmentation use standard recipes without extensive hyperparameter tuning, so transfer performance may be underestimated.
- Mamba-based models show lower accuracy than some ViT and CNN baselines in the tables (e.g., MaxViT-B 84.9%, FasterViT-2 84.2%) on certain trade-offs.
Experimental setup
Datasets: ImageNet-1K ImageNet-21K MS COCO ADE20K
Benchmarks: ImageNet-1K MS COCO ADE20K
Metrics: Top-1 accuracy image throughput box AP mask AP mIoU GFLOPs
Baselines: ConvNeXt Swin Transformer SwinV2 VMamba Vim EfficientVMamba FasterViT MaxViT NextViT Focal Transformer SiMBA DeiT
Builds on: Mamba Vision Transformer Swin Transformer ConvNeXt VMamba
Closest papers by meaning
- Demystify Mamba in Vision: A Linear Attention Perspective (NeurIPS 2024 · similarity 0.90)
- MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone's Potential with Masked Autoregressive Pretraining (CVPR 2025 · similarity 0.90)
- MambaOut: Do We Really Need Mamba for Vision? (CVPR 2025 · similarity 0.89)
- Mamba-Reg: Vision Mamba Also Needs Registers (CVPR 2025 · similarity 0.89)
- Vision Mamba Mender (NeurIPS 2024 · similarity 0.89)
- MobileMamba: Lightweight Multi-Receptive Visual Mamba Network (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.