Research map / Architectures and efficiency / Attention and transformers
Mamba-Adaptor: State Space Model Adaptor for Visual Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mamba-Adaptor, a plug-and-play module with a learnable memory-selection step (Adaptor-T) and multi-scale depthwise convolution (Adaptor-S) that augments Mamba/VMamba for visual backbones, boosting, and transfer learning.
Problem
Vision Mamba models underperform on visual tasks due to causal computation without global context, long-range forgetting of hidden states, and loss of 2D spatial structure when flattening images into sequences.
Approach
Adds a learnable memory-selection module that aggregates predicted most-forgotten hidden states into the current hidden state (Adaptor-T), and multi-scale dilated depthwise convolutions applied to the reshaped 2D output (Adaptor-S), inserted into an optimized selective-scan solver via identity C and zero D matrices.
Key results
- Mamba-Adaptor-b2 reaches 83.0% top-1 on ImageNet-1K with 32M params and 5.4G FLOPs, exceeding VMamba-T by 0.2 points (text states 82.9 in one place).
- Mamba-Adaptor-b1 with Mask R-CNN 1x reaches 43.2 box AP on COCO, 3.9 above EffVMamba-S.
- As a booster on VMamba-T, Adaptor adds about 0.1% top-1 accuracy on ImageNet with 3.2% extra parameters, and 0.2% on VMamba-B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add learnable state selection instead of static state selection (ablation) | Mamba-Adaptor · ImageNet-1K val | Top-1 accuracy: 77.9 -> 78.2 | Helped |
| Add learnable state selection instead of static state selection (ablation) | Mamba-Adaptor · COCO object detection | AP b: 42.3 -> 42.9 | Helped |
| Add a second depthwise-convolution scale to Adaptor-S (ablation) | Mamba-Adaptor · COCO instance segmentation | AP m: 44.4 -> 44.8 | Helped |
| Add static state selection (ablation) | Mamba-Adaptor · ImageNet-1K val | Top-1 accuracy: 77.2 -> 77.6 | Helped |
| Add 1-scale Adaptor-S (ablation) | Mamba-Adaptor · ImageNet-1K val | Top-1 accuracy: 77.2 -> 77.5 | Helped |
| Use zero instead of random initialization, with sequential insertion (ablation) | Mamba-Adaptor transfer-learning setup · CIFAR-100 | accuracy: 62.34 -> 85.33 | Helped |
| Use zero instead of random initialization, with sequential insertion (ablation) | Mamba-Adaptor transfer-learning setup · SVHN | accuracy: 54.32 -> 92.43 | Helped |
| Use zero instead of random initialization, with sequential insertion (ablation) | Mamba-Adaptor transfer-learning setup · Food-101 | accuracy: 65.12 -> 83.43 | Helped |
| Use parallel instead of sequential insertion (ablation) | Mamba-Adaptor transfer-learning setup · CIFAR-100 | accuracy: 85.33 -> 77.23 | Hurt |
| Use parallel instead of sequential insertion (ablation) | Mamba-Adaptor transfer-learning setup · SVHN | accuracy: 92.43 -> 67.98 | Hurt |
| Use parallel instead of sequential insertion (ablation) | Mamba-Adaptor transfer-learning setup · Food-101 | accuracy: 83.43 -> 78.32 | Hurt |
| Add weight-sharing coefficients (ablation) | Mamba-Adaptor transfer-learning setup · CIFAR-100 | accuracy: 77.23 -> 86.82 | Helped |
Limitations
- Scaling of Mamba-Adaptor as a visual backbone remains open
- Applications to other vision Mamba variants are left as future work
- Transfer-learning gap to full fine-tuning remains (about 1-12 points on the three benchmarks)
Experimental setup
Datasets: ImageNet-1K COCO CIFAR-100 SVHN Food-101
Benchmarks: ImageNet-1K COCO CIFAR-100 SVHN Food-101
Metrics: top-1 accuracy box AP mask AP parameters FLOPs
Baselines: Vim-T LocalViM-T VMamba-T LocalVMamba-T EffVMamba-S Swin-T ConvNeXt-T VPT
Builds on: Mamba VMamba Vision Mamba VPT Mask R-CNN
Closest papers by meaning
- Vision Mamba Mender (NeurIPS 2024 · similarity 0.92)
- Demystify Mamba in Vision: A Linear Attention Perspective (NeurIPS 2024 · similarity 0.91)
- MambaOut: Do We Really Need Mamba for Vision? (CVPR 2025 · similarity 0.91)
- Spatial-Mamba: Effective Visual State Space Models via Structure-Aware State Fusion (ICLR 2025 · similarity 0.90)
- Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model (NeurIPS 2024 · similarity 0.89)
- MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.