Research map / Architectures and efficiency / Attention and transformers
Adventurer: Optimizing Vision Mamba Architecture Designs for Efficiency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Adventurer, a causal (uni-directional) image model that treats image patches as a 1D sequence, using a heading average token and inter-layer flipping to achieve linear-complexity vision encoding that matches ViT accuracy with much higher throughput.
Problem
Vision transformers have quadratic cost in sequence length, making high-resolution and fine-grained image modeling expensive; existing vision Mamba models are slow due to multi-directional scanning.
Approach
Treat images as patch token sequences processed by a single uni-directional causal (Mamba-2) scan, fixing the start-of-sequence information deficit with a heading average pooling token recomputed each layer and flipping token order between layers, instead of multi-way scanning.
Key results
- Adventurer-Base reaches 84.3% ImageNet top-1 at 448px with 216 img/s training throughput, 3.8x faster than Vim and 6.2x faster than DeiT to reach the same accuracy.
- Adventurer-Base/P8 (448px, sequence ~3,000 tokens) reaches 84.8% ImageNet top-1 at 57 img/s.
- Adventurer-Base achieves 46.6 mIoU on ADE20k at 0.86x DeiT-Base latency, and 48.4 box AP on COCO with Cascade Mask R-CNN (Tiny/Small/Base variants reported).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add heading average and inter-layer flipping (ablation) | Causal DeiT-Small · ImageNet-1k | accuracy: 78.8 -> 79.9 (+1.1) | Helped |
| add heading average (ablation) | Causal DeiT-Small · ImageNet-1k | accuracy: 78.8 -> 79.1 (+0.3) | Helped |
| add inter-layer flipping (ablation) | Causal DeiT-Small · ImageNet-1k | accuracy: 78.8 -> 79.6 (+0.8) | Helped |
| add heading average and inter-layer flipping (ablation) | Adventurer-Small · ImageNet-1k | accuracy: 80.3 -> 81.8 (+1.5) | Helped |
| add heading average (ablation) | Adventurer-Small · ImageNet-1k | accuracy: 80.3 -> 80.8 (+0.5) | Helped |
| add inter-layer flipping (ablation) | Adventurer-Small · ImageNet-1k | accuracy: 80.3 -> 81.2 (+0.9) | Helped |
| use SwiGLU MLP channel mixers instead of doubled Mamba layers (ablation) | Adventurer · ImageNet-1k | accuracy: 78.0 / 81.6 / 82.3 -> 78.2 / 81.8 / 82.6 | Helped |
| use SwiGLU MLP channel mixers instead of simple MLP channel mixers (ablation) | Adventurer · ImageNet-1k | accuracy: 78.0 / 81.7 / 82.4 -> 78.2 / 81.8 / 82.6 | Helped |
| use multiple heading tokens rather than one (ablation) | Adventurer-Small · ImageNet-1k | accuracy: 79.5 / 79.5 -> 79.3 | Hurt |
| use a duplicated class token as the heading token instead of an average token (ablation) | Adventurer-Small · ImageNet-1k | accuracy: 81.4 -> 81.8 | Hurt |
| use a learned heading token instead of an average token (ablation) | Adventurer-Small · ImageNet-1k | accuracy: 81.5 -> 81.8 | Hurt |
| recalculate the heading average token at the beginning of each layer (ablation) | Adventurer-Small · ImageNet-1k | accuracy: 81.4 -> 81.8 | Helped |
Surprising findings
- Naive causal ViT loses about 1.1% ImageNet accuracy, yet heading average plus flipping closes the gap entirely to standard fully-visible ViTs.
- Multi-way scanning (2-way, 4-way) adds only +0.06% and +0.11% accuracy while raising latency 1.9x and 3.7x.
Limitations
- Position encoding designed for causal models is left to future work; learnable absolute positions are used.
- Gains on classification are modest at standard resolution compared with some baselines; the main advantage is efficiency at high resolution.
Experimental setup
Datasets: ImageNet-1k ADE20k COCO 2017
Benchmarks: ImageNet-1k ADE20k COCO 2017
Metrics: top-1 accuracy mIoU box AP mask AP training throughput GPU memory latency
Baselines: DeiT Vim MambaReg ConvNeXt VMamba EfficientNet ResNet
Builds on: Mamba Mamba-2 Vision Transformer (ViT) Vim MambaReg
Closest papers by meaning
- Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models (NeurIPS 2024 · similarity 0.87)
- Stochastic Layer-Wise Shuffle for Improving Vision Mamba Training (ICML 2025 · similarity 0.86)
- Vision Mamba Mender (NeurIPS 2024 · similarity 0.86)
- Mamba-Reg: Vision Mamba Also Needs Registers (CVPR 2025 · similarity 0.85)
- iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.85)
- HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.