Research map / Architectures and efficiency / Attention and transformers
MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone's Potential with Masked Autoregressive Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Masked Autoregressive Pretraining (MAP), which combines local MAE masking with row-wise autoregressive reconstruction to pretrain hybrid Mamba-Transformer vision backbones (HybridNet), also improving pure Mamba and pure Transformer models.
Problem
Pretraining hybrid Mamba-Transformer vision backbones is unresolved, since MAE suits Transformers and autoregressive pretraining suits Mamba, and neither transfers well to both components.
Approach
Randomly mask image tokens (50%), then reconstruct the image row by row autoregressively with a Transformer decoder, where tokens within each row are predicted simultaneously; this gives local bidirectional MAE-style modeling for Transformer blocks and AR-style context modeling aligned with Mamba's scan order.
Key results
- HybridNet-B + MAP reaches 84.9% top-1 on ImageNet-1K at 224 resolution (85.5% at 384), versus 83.9% for MAE and 83.8% for AR on the same backbone.
- HybridNet-S + MAP achieves 46.9 mIoU on ADE20K, up from 45.6 without pretraining.
- HybridNet with MAP reaches 95.9% on ModelNet40 with voting, above Mamba3D with Point-MAE (95.4%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MAP instead of MAE pretraining | HybridNet-B · ImageNet-1K val, 224² | top-1 accuracy (%): 83.9 -> 84.9 | Helped |
| Use MAP instead of AR pretraining | HybridNet-B · ImageNet-1K val, 224² | top-1 accuracy (%): 83.8 -> 84.9 | Helped |
| Use MAP instead of AR pretraining | MambaR-B · ImageNet-1K val, 224² | top-1 accuracy (%): 83.7 -> 84.0 | Helped |
| Use MAP instead of MAE pretraining | ViT-L/16 · ImageNet-1K val, 224² | top-1 accuracy (%): 85.9 -> 86.1 | Helped |
| Use MAP pretraining | HybridNet-S · ADE20K val | mIoU: 45.6 -> 46.9 | Helped |
| Use MAP pretraining | HybridNet-Ti · COCO val | APbox: 45.9 -> 46.4 | Helped |
| Use MAP pretraining | HybridNet-Ti · COCO val | APmask: 39.2 -> 39.8 | Helped |
| Use MAP pretraining | HybridNet w/o voting · ModelNet40 | accuracy: 93.5 -> 95.4 | Helped |
| Align AR pretraining order with Mamba scanning order (ablation) | Vim · ImageNet-1K | accuracy: 79.7 -> 82.6 | Helped |
| Use 40 masked tokens (20%) for AR pretraining (ablation) | Vim · ImageNet-1K | accuracy: 81.7 -> 82.6 | Helped |
| Use random rather than sequential or diagonal masking (ablation) | HybridNet-B MAP · ImageNet-1K | accuracy: 84.0 -> 84.9; 83.8 -> 84.9 | Helped |
| Use the proposed row-wise decoder mask rather than AR decoding (ablation) | HybridNet-B MAP · ImageNet-1K | accuracy: 83.7 -> 84.9 | Helped |
Surprising findings
- MAP with a 50% masking ratio outperforms the 75% ratio optimal for MAE, and AR's best ratio on Mamba was 20%.
- On pure Transformers (ViT-B), MAP only matches MAE rather than surpassing it, and the hybrid with MAP still trails a pure Transformer with MAE under the same settings.
Limitations
- The row-wise autoregressive order is chosen to match row-first Mamba scanning; the paper leaves other clustering/scan orders for future work.
- Hybrid with MAP still struggles to surpass a pure Transformer with MAE under the same settings.
- Design of the hybrid architecture is fixed empirically (MMMTMMMT) and not studied in depth.
Experimental setup
Datasets: ImageNet-1K ADE20K COCO 2017 ShapeNet ModelNet40
Benchmarks: ImageNet-1K ADE20K COCO 2017 ModelNet40
Metrics: top-1 accuracy mIoU APbox APmask overall accuracy
Baselines: MAE AR (ARM) Contrastive learning (MoCov2) ViT Vim MambaR VMamba MambaVision Point-MAE Point-BERT Mamba3D
Builds on: Masked Autoencoders (MAE) ARM autoregressive pretraining for Mamba Vision Mamba MambaR Mamba3D
Closest papers by meaning
- MambaVision: A Hybrid Mamba-Transformer Vision Backbone (CVPR 2025 · similarity 0.90)
- Autoregressive Pretraining with Mamba in Vision (ICLR 2025 · similarity 0.88)
- The Mamba in the Llama: Distilling and Accelerating Hybrid Models (NeurIPS 2024 · similarity 0.87)
- Binarized Mamba-Transformer for Lightweight Quad Bayer HybridEVS Demosaicing (CVPR 2025 · similarity 0.85)
- MobileMamba: Lightweight Multi-Receptive Visual Mamba Network (CVPR 2025 · similarity 0.85)
- MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methods (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.