Research map / Architectures and efficiency / Attention and transformers
Mamba-Reg: Vision Mamba Also Needs Registers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies high-norm artifact tokens in Vision Mamba and proposes Mamba® (Mamba-Reg), which evenly inserts register tokens into the Vim sequence and recycles them via a reduce-and-concat head for prediction, improving ImageNet accuracy and scaling to 341M parameters.
Problem
Vision Mamba models produce severe high-norm artifact tokens in low-information background regions, which degrade feature maps and hinder performance and scaling.
Approach
Adapt ViT register tokens to the unidirectional Mamba scan by inserting registers evenly throughout the input sequence rather than at one end, and by concatenating (after a linear reduction) all register outputs as the final global representation instead of discarding them.
Key results
- Mamba®-B reaches 83.0% ImageNet top-1 at 224x224 versus 81.8% for Vim-B, and 84.5% at 384x384 with the 341M Large model.
- Mamba®-S achieves 45.3% mIoU on ADE20k versus 44.9% for Vim-S; Mamba®-L reaches 49.1% mIoU.
- Ablation: evenly inserted registers with reduce-and-concat head give 83.0% vs 82.1% when all registers are placed at the beginning and 81.7-82.2% with other final-prediction protocols.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Introduce registers into Vision Mamba with even spacing and a reduce-and-concatenate prediction head | Vision Mamba · ImageNet-1k | accuracy (%): 81.8 -> 83.0 | Helped |
| Introduce registers into Vision Mamba | Vision Mamba · ADE20k validation | mIoU (%): 44.9 -> 45.3 | Helped |
| Use reduce-and-concatenate register prediction instead of one register or the mean, with registers at the beginning, middle, or evenly spaced (ablation) | Mamba®-B · ImageNet-1k | accuracy (%): 81.3 -> 82.1; 81.8 -> 82.6; 81.7 -> 83.0; 81.4 -> 82.1; 82.0 -> 82.6; 82.2 -> 83.0 | Helped |
| Evenly distribute registers instead of placing them all at the beginning or middle (ablation) | Mamba®-B with reduce-and-concatenate prediction · ImageNet-1k | accuracy (%): 82.1 -> 83.0; 82.6 -> 83.0 | Helped |
| Use 12 registers instead of 1, 3, 6, or 24 (ablation) | Mamba®-B · ImageNet-1k | accuracy (%): 81.8; 82.0; 82.8; 82.8; 83.0; 82.6 | Mixed |
| Increase output dimension without adding registers (ablation) | Vim-B · ImageNet-1k | accuracy (%): 81.8 -> 82.0 | Helped |
Surprising findings
- Vision Mamba artifacts appear even in the tiny model and grow with depth, reaching norms above 4000 by layer 23, more severe than in ViTs.
- Registers in Vim serve the quantitative performance role strongly, unlike the ViT register finding that they mainly improve feature maps.
Limitations
- Registers at the final layer and register-head design require tuning of count and reduction factor per model size.
- Scaling beyond 341M parameters and other tasks beyond classification and segmentation are not evaluated.
Experimental setup
Datasets: ImageNet-1k ADE20k
Benchmarks: ImageNet-1k ADE20k
Metrics: top-1 accuracy mIoU throughput memory overhead feature norm distribution
Baselines: Vim-T Vim-S VMamba-T VMamba-S VMamba-B LocalVim-S PlainMamba-L3 DeiT-S DeiT-B ConvNeXt-B
Builds on: Vision Transformers Need Registers Vision Mamba (Vim) DeiT training recipe UperNet
Closest papers by meaning
- MambaOut: Do We Really Need Mamba for Vision? (CVPR 2025 · similarity 0.91)
- MambaVision: A Hybrid Mamba-Transformer Vision Backbone (CVPR 2025 · similarity 0.89)
- Vision Mamba Mender (NeurIPS 2024 · similarity 0.89)
- Demystify Mamba in Vision: A Linear Attention Perspective (NeurIPS 2024 · similarity 0.89)
- Register and [CLS] tokens induce a decoupling of local and global features in large ViTs (NeurIPS 2025 · similarity 0.88)
- QuadMamba: Learning Quadtree-based Selective Scan for Visual State Space Model (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.