Research map / Vision-language and multimodal / CLIP and zero-shot
Scalable Pre-training of Large Autoregressive Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AIM, ViT models pre-trained with a prefix-causal autoregressive pixel-regression objective on 2B uncurated web images (DFN-2B+), showing scaling with capacity and data up to 7B parameters with frozen-trunk attentive probing.
Problem
Whether the scaling success of autoregressive LLM pre-training transfers to visual representation learning, i.e., whether autoregressive pre-training of large ViTs yields strong, scalable visual features.
Approach
Pre-train ViTs with a next-patch autoregressive objective (normalized pixel MSE) using prefix attention so the trunk can be used bidirectionally downstream, plus a heavy MLP prediction head; attentive probing with a frozen trunk. No stability tricks are needed to scale to 7B parameters.
Key results
- AIM-7B with a frozen trunk reaches 84.0% top-1 on ImageNet-1k (extracting from the 20th layer) and averages 79.6 over 15 benchmarks.
- Average attentive-probe accuracy over 15 benchmarks rises monotonically with capacity: 75.6 (0.6B), 76.9 (1B), 77.6 (3B), 77.8 (7B, last layer).
- LoRA (rank 8) fine-tuning of AIM-7B reaches 86.3% on ImageNet-1k, up from 82.4% frozen last-layer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model capacity from AIM-0.6B to AIM-7B | AIM · 15 image recognition benchmarks | average accuracy: 75.6 -> 77.8 | Helped |
| Extract features from the 20th layer rather than the last layer (ablation) | AIM-7B · ImageNet-1k | top-1 accuracy: 82.4 -> 84.0 | Helped |
| Mix DFN-2B with IN-1k during pre-training (ablation) | AIM-0.6B · 15 benchmarks | attentive probe accuracy: 74.5 -> 75.6 | Helped |
| Use normalized pixel targets instead of raw pixels, KMeans, or dVAE tokens (ablation) | AIM-0.6B · ImageNet-1k | attentive probe accuracy: 76.2, 75.9, 74.5 -> 78.2 | Helped |
| Use raster traversal instead of spiral, checkerboard, or random order (ablation) | AIM-0.6B · ImageNet-1k | attentive probe accuracy: 76.3, 76.0, 75.7 -> 77.4 | Helped |
| Use crop scale 0.4 rather than 0.08 or 1.0 (ablation) | AIM-0.6B · ImageNet-1k | attentive probe accuracy: 77.7, 63.5 -> 78.2 | Helped |
| Use prefix attention during pre-training and bidirectional attention at inference instead of causal attention in both (ablation) | AIM-0.6B · ImageNet-1k | attentive probe accuracy: 77.4 -> 78.2 | Helped |
| Replace the MLP prediction head with a Transformer head (ablation) | AIM-0.6B · ImageNet-1k | attentive probe accuracy: 78.2 -> 78.5 | Helped |
| Add an MLP prediction head instead of using no head (ablation) | AIM-0.6B · ImageNet-1k | attentive probe accuracy: 75.4 -> 78.2 | Helped |
| Use a wider rather than deeper architecture (ablation) | AIM-0.6B · ImageNet-1k | attentive probe accuracy: 77.9 -> 78.2 | Helped |
| Increase MLP head width from 512 to 2048 (ablation) | AIM-0.6B · ImageNet-1k | attentive probe accuracy: 77.7 -> 78.2 | Helped |
| Increase MLP head depth from 6 to 12 blocks (ablation) | AIM-0.6B · ImageNet-1k | attentive probe accuracy: 76.2 -> 78.2 | Helped |
Surprising findings
- Features from an intermediate layer (about two-thirds depth) outperform the last layer for all AIM variants, consistent with the generative objective differing from discriminative tasks.
- Raster order substantially outperforms spiral, checkerboard and random autoregression orders, and a per-patch normalized pixel target beats k-means and dVAE discrete targets.
Limitations
- Contrastive methods still give stronger representations per model size than generative approaches like AIM and MAE.
- Best DINOv2 results rely on higher-resolution inputs, which AIM does not use, so AIM falls behind DINOv2 on average.
- Scaling laws are not fully characterized; the authors defer further investigation of scaling law behavior to future work.
Experimental setup
Datasets: DFN-2B DataComp 12.8B ImageNet-1k iNaturalist 2018 CIFAR-10 CIFAR-100 Food101 DTD Oxford Pets Stanford Cars iWildCam Camelyon17
Benchmarks: ImageNet-1k iNaturalist 2018 CIFAR-10 CIFAR-100 Food101 DTD Pets Cars iWildCam Camelyon17 PCAM RxRx1
Metrics: top-1 accuracy attentive probe accuracy linear probe accuracy validation pre-training loss
Baselines: DINO iBOT DINOv2 BEiT MAE BigGAN
Builds on: iGPT T5 prefixLM Vision Transformer BEiT MAE
Closest papers by meaning
- Data-efficient Large Vision Models through Sequential Autoregression (ICML 2024 · similarity 0.92)
- Multimodal Autoregressive Pre-training of Large Vision Encoders (CVPR 2025 · similarity 0.92)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.89)
- Sequential Modeling Enables Scalable Learning for Large Vision Models (CVPR 2024 · similarity 0.88)
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.88)
- MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.