Research map / Vision-language and multimodal / CLIP and zero-shot
Multimodal Autoregressive Pre-training of Large Vision Encoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AIMV2, a family of 0.3B–3B vision encoders pre-trained with a multimodal autoregressive objective that reconstructs image patches and text tokens through a causal multimodal decoder.
Problem
Generative pre-training of vision encoders lags discriminative (contrastive) methods in performance and parameter efficiency, and existing vision-language encoders are hard to train and scale; the paper seeks a simple, scalable generalist vision encoder aligned with LLM-based multimodal pipelines.
Approach
Extends the AIM autoregressive image pre-training with a joint causal multimodal decoder that first regresses image patches (pixel MSE) and then predicts caption tokens (cross-entropy), using prefix attention in the vision encoder, so every patch and text token provides supervision.
Key results
- AIMV2-3B at 448px reaches 89.5% ImageNet-1k top-1 with a frozen trunk (attentive probe), above DINOv2 ViT-g (87.2%) and SigLIP So400m (87.3%).
- AIMV2 outperforms DINOv2 on open-vocabulary detection (LVIS 31.6 vs 30.8) and most referring expression benchmarks in the grounding evaluation.
- AIMV2 uses about 12B image-text samples versus 40B for DFN-CLIP and SigLIP, while matching or exceeding them on several recognition and multimodal benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the multimodal autoregressive objective instead of captioning-only training (ablation) | AIMV2-L · ImageNet-1k | top-1 accuracy: 85.1 -> 85.6 | Helped |
| Use the multimodal autoregressive objective instead of captioning-only training (ablation) | AIMV2-L · TextVQA | score: 34.4 -> 37.5 | Helped |
| Use prefix attention instead of bidirectional attention (ablation) | Captioning-only model · VQAv2 | score: 76.2 -> 76.8 | Helped |
| Add the captioning objective (ablation) | AIMV1 · ImageNet-1k | top-1 accuracy: 72.0 -> 85.6 | Helped |
| Increase CLIP batch size from 8k to 16k (ablation) | CLIP · TextVQA | score: 24.6 -> 26.3 | Helped |
| Vary image loss weight ε from 0.2 to 0.6 (ablation) | AIMV2-L · ImageNet-1k | top-1 accuracy: 85.6 -> 85.6 | No clear effect |
| Use one joint decoder instead of separate modality decoders (ablation) | AIMV2-L · VQAv2 | score: 77.1 -> 76.9 | No clear effect |
| Increase decoder width from 512 to 1024 (ablation) | AIMV2-L · VQAv2 | score: 76.2 -> 76.9 | Helped |
| Increase decoder depth from 8 to 12 (ablation) | AIMV2-L · VQAv2 | score: 76.7 -> 76.9 | Helped |
| Scale decoder depth from 12 to 16 (ablation) | AIMV2-L · TextVQA | score: 37.5 -> 36.6 | Hurt |
| Scale AIMV2 from ViT-L to ViT-3B | AIMV2 · VQAv2 | score: 79.7 -> 80.9 | Helped |
| Use AIMV2 instead of OAI CLIP | ViT-L multimodal instruction tuning · TextVQA | score: 47.5 -> 53.6 | Helped |
Surprising findings
- Optimal model size depends on compute budget: larger models underperform smaller ones when severely undertrained.
- Zero-shot IN-1k with LiT is weaker than SigLIP, which is trained with 40B pairs.
Limitations
- Zero-shot recognition via LiT lags dedicated models such as SigLIP trained on 40B pairs.
- Pre-training relies on a proprietary HQITP dataset, so full reproduction is limited.
- DINOv2 remains stronger on iNaturalist and fMoW.
Experimental setup
Datasets: DFN-2B COYO HQITP Llava SFT mixture Cambrian 7M ImageNet-1k Objects365 Flickr30k Entities GQA COCO RefCOCO
Benchmarks: ImageNet-1k iNaturalist-18 CIFAR-10 CIFAR-100 Food101 DTD Pets Cars Camelyon17 PCAM RxRx1 EuroSAT
Metrics: top-1 accuracy mean average precision precision@1 CIDEr MME perception score
Baselines: MAE AIMV1 DINOv2 OAI CLIP DFN-CLIP SigLIP CapPa MM-Grounding-DINO
Builds on: AIM Llava instruction tuning LiT MM1 Cambrian
Closest papers by meaning
- Scalable Pre-training of Large Autoregressive Image Models (ICML 2024 · similarity 0.92)
- Data-efficient Large Vision Models through Sequential Autoregression (ICML 2024 · similarity 0.89)
- MoVA: Adapting Mixture of Vision Experts to Multimodal Context (NeurIPS 2024 · similarity 0.88)
- iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.88)
- UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.88)
- Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.