Research map / Architectures and efficiency / Attention and transformers
Sample- and Parameter-Efficient Auto-Regressive Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes XTRA, a pixel-reconstruction auto-regressive ViT encoder-decoder with Block Causal Masking (k×k token blocks predicted in raster order), improving sample and parameter efficiency over AIM and iGPT.
Problem
Auto-regressive image models such as iGPT and AIM are sample- and parameter-inefficient, requiring very large models and datasets to reach competitive self-supervised representation quality.
Approach
Replace the standard token-level causal mask with a Block Causal Mask where tokens attend freely within a k×k block and causally to preceding blocks in raster order, and reconstruct the pixels of the next block with an MSE loss, so each autoregressive step captures larger, more semantic image regions.
Key results
- XTRA-H/14 trained on 13.1M samples reaches 76.2 average accuracy across 15 benchmarks vs 75.6 for AIM-0.6B (DFN-2B+) and 74.5 for AIM-0.6B (DFN-2B), with 152x fewer samples.
- XTRA-B/16 (85M params) on ImageNet-1K achieves 70.2% linear probing and 76.8% attentive probing at 800 epochs, versus iGPT-L 65.2% LIN (1.36B) and AIM-0.6B 73.5% ATT (0.63B).
- Block size ablation: 64x64 pixel blocks improve ImageNet attentive probing accuracy by 3.0% over the single-token AIM-style setting (67.6 vs 64.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use block causal masking instead of standard single-token autoregression | XTRA-H · 15 image recognition benchmarks | average accuracy: 74.5 -> 76.2 | Helped |
| use block causal masking instead of standard single-token autoregression | XTRA-B · ImageNet-1K | attentive probing accuracy: 73.5 -> 76.8 | Helped |
| increase block size from 16×16 to 64×64 pixels (ablation) | XTRA · ImageNet-1K | attentive probing accuracy: 64.6 -> 67.6 | Helped |
| predict 2 blocks instead of 1 with 16×16-pixel blocks (ablation) | XTRA · ImageNet-1K | attentive probing accuracy: 64.6 -> 65.1 | Helped |
| predict 2 blocks instead of 1 with 32×32-pixel blocks (ablation) | XTRA · ImageNet-1K | attentive probing accuracy: 67.4 -> 67.3 | No clear effect |
| predict 2 blocks instead of 1 with 64×64-pixel blocks (ablation) | XTRA · ImageNet-1K | attentive probing accuracy: 67.6 -> 67.4 | No clear effect |
| use L2 loss instead of L1 loss (ablation) | XTRA · ImageNet-1K | attentive probing accuracy: 66.6 -> 67.6 | Helped |
| use raster block order instead of fixed random order (ablation) | XTRA · ImageNet-1K | attentive probing accuracy: 55.7 -> 67.6 | Helped |
| vary decoder depth from 1 to 16 blocks (ablation) | XTRA · ImageNet-1K | attentive probing accuracy: 67.4, 67.9, 67.6, 67.6, 67.8 | No clear effect |
| increase decoder width from 192 to 768 (ablation) | XTRA · ImageNet-1K | attentive probing accuracy: 67.1 -> 69.6 | Helped |
Surprising findings
- A fixed random block order drops ImageNet attentive accuracy to 55.7% versus 67.6% for raster order, showing the sequence ordering matters greatly.
Limitations
- Some reconstructed blocks lack entire object details due to limited context, as noted in qualitative results
- ImageNet-21K subset filtered to 13.1M samples due to broken URLs, limiting exact reproducibility
- Comparisons to contrastive methods are restricted to single-view settings for fairness; generative models still lag multi-view methods such as iBOT
Experimental setup
Datasets: ImageNet-21K (filtered to 13.1M samples) ImageNet-1K
Benchmarks: ImageNet-1K iNaturalist 2018 CIFAR-10 CIFAR-100 Food101 DTD Pets Cars iWildCam Camelyon17 PCAM RxRx1
Metrics: top-1 accuracy linear probing accuracy attentive probing accuracy
Baselines: AIM-0.6B iGPT-L MAE CAE data2vec iBOT DINO BEiT MoCo-v3 I-JEPA SimMIM StoP
Builds on: AIM iGPT MAE Vision Transformer
Closest papers by meaning
- xT: Nested Tokenization for Larger Context in Large Images (ICML 2024 · similarity 0.85)
- Improved Baselines with Representation Autoencoders (NeurIPS 2026 · similarity 0.85)
- Data-efficient Large Vision Models through Sequential Autoregression (ICML 2024 · similarity 0.85)
- SATA: Spatial Autocorrelation Token Analysis for Enhancing the Robustness of Vision Transformers (CVPR 2025 · similarity 0.85)
- MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modeling (CVPR 2025 · similarity 0.85)
- LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.