Research map / Architectures and efficiency / Attention and transformers
Autoregressive Pretraining with Mamba in Vision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ARM, autoregressive pretraining on ImageNet-1K for a Mamba-based vision backbone (MambaMLP) that predicts clusters of 64x64 patches in row-first forward order, reaching 83.2% (base) and 85.0% (huge) ImageNet top-1.
Problem
Supervised training of Mamba vision backbones transfers poorly and fails to scale (Vim plateaus or collapses at huge size); self-supervised pretraining for Mamba in vision is unexplored.
Approach
Apply autoregressive next-cluster prediction to Mamba: group spatially neighboring patches into larger prediction units (64x64 clusters), use a row-first forward raster order, and use a MambaMLP block with a single scan during pretraining that matches unidirectional recurrence, then switch to 4 scans for finetuning.
Key results
- ARM-B reaches 83.2% ImageNet top-1, 2.0% above supervised MambaMLP-B (81.2%) and 2.0% above Vim-B.
- ARM-H reaches 85.0% top-1 (85.5% at 384x384), the largest Vision Mamba reported, while supervised Vim-H collapsed.
- ARM-B outperforms Vim-B by 4.4% on ImageNet-A, 2.9% on ImageNet-R and 3.3% on ImageNet-Sketch (robustness table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ARM autoregressive pretraining instead of supervised training | MambaMLP-B · ImageNet-1K | Top-1 accuracy: 81.2 -> 83.2 | Helped |
| Apply ARM autoregressive pretraining instead of supervised training | MambaMLP-L · ImageNet-1K | Top-1 accuracy: 81.4 -> 84.5 | Helped |
| Apply ARM autoregressive pretraining instead of supervised training | MambaMLP-H · ImageNet-1K | Top-1 accuracy | Helped |
| Increase finetuning input size from 224² to 384² | ARM-B · ImageNet-1K | Top-1 accuracy: 83.2 -> 84.2 | Helped |
| Increase finetuning input size from 224² to 384² | ARM-L · ImageNet-1K | Top-1 accuracy: 84.5 -> 85.1 | Helped |
| Increase finetuning input size from 224² to 384² | ARM-H · ImageNet-1K | Top-1 accuracy: 85.0 -> 85.5 | Helped |
| Replace supervised pretraining with ARM pretraining (ablation) | Vim-B · ImageNet-1K | Top-1 accuracy: 81.2 -> 82.2 | Helped |
| Use ARM autoregressive pretraining instead of supervised training (ablation) | MambaMLP-B · ImageNet-1K, 300-epoch pretraining | Top-1 accuracy: 81.2 -> 82.5 | Helped |
| Use MAE pretraining instead of supervised training (ablation) | MambaMLP-B · ImageNet-1K, 300-epoch pretraining | Top-1 accuracy: 81.2 -> 81.6 | Helped |
| Use contrastive pretraining instead of supervised training (ablation) | MambaMLP-B · ImageNet-1K, 300-epoch pretraining | Top-1 accuracy: 81.2 -> 81.4 | Helped |
| Use random cluster order instead of predefined order (ablation) | ARM-B · ImageNet-1K, 300-epoch pretraining | Top-1 accuracy: 82.5 -> 81.5 | Hurt |
| Vary predefined cluster ordering among row-first and column-first directions (ablation) | ARM-B · ImageNet-1K, 300-epoch pretraining | Top-1 accuracy: 82.5, 82.3, 82.5, 82.4 | No clear effect |
Surprising findings
- Random cluster ordering drops accuracy to 81.5%, far below predefined raster orders (82.5%), while the four raster variants differ by only 0.2%.
- Pixel-level iGPT-style autoregression (79.8%) underperforms even the supervised baseline (81.2%).
Limitations
- Gains from pretraining over supervised Vim are smaller at some sizes and ARM-B slightly trails VMamba-B at 224 resolution (0.7%).
- Architecture is a relatively naive ViT-like Mamba design (MambaMLP) rather than a novel vision-specific architecture.
- Only ImageNet-1K is used for pretraining, so scaling with larger data is not explored.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet-1K ImageNet-V2 ImageNet-Real ImageNet-A ImageNet-R ImageNet-Sketch
Metrics: top-1 accuracy throughput (imgs/s) training cost (h)
Baselines: Vim VMamba-B MambaMLP supervised MAE contrastive learning RegNetY-16G DeiT-B iGPT-style pixel autoregression
Builds on: Mamba iGPT MAE Vision Mamba VMamba
Closest papers by meaning
- MambaOut: Do We Really Need Mamba for Vision? (CVPR 2025 · similarity 0.91)
- Vision Mamba Mender (NeurIPS 2024 · similarity 0.89)
- MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone's Potential with Masked Autoregressive Pretraining (CVPR 2025 · similarity 0.88)
- Demystify Mamba in Vision: A Linear Attention Perspective (NeurIPS 2024 · similarity 0.87)
- Mamba-Reg: Vision Mamba Also Needs Registers (CVPR 2025 · similarity 0.87)
- Scalable Pre-training of Large Autoregressive Image Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.