Research map / Image generation / Autoregressive and latent generation
Denoising Autoregressive Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DARL, a decoder-only ViT that predicts image patches autoregressively, trained with MSE or a denoising patch-diffusion objective, evaluated as a visual representation learner on ImageNet, VTAB and COCO.
Problem
Representation learning for vision is usually separated from image generation; the paper asks whether an autoregressive generative pre-training approach can match masked prediction representation learners while also enabling generation.
Approach
Autoregressive patch prediction with a causal ViT and 2D RoPE, where the patch decoder is either a linear MSE head or a single-block denoising Transformer decoder trained with a diffusion ELBO-style objective using a tuned Beta noise schedule on the noise level gamma.
Key results
- DARL (DIFF) ViT-L16 reaches 84.9% ImageNet top-1 under fine-tuning (800 epochs pre-training, 90 fine-tuning), versus MAE 85.9%.
- Optimal diffusion noise schedule (Beta a=0.03, b=1) gives 83.6% vs 82.9% for noise prediction at 100 epochs pre-training, with DARL (DIFF) outperforming MSE after longer training.
- On COCO with ViT-L16 and Cascade Mask R-CNN, DARL (DIFF) gets 56.4 AP vs 57.2 for reproduced MAE and 54.1 for supervised.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace MSE with the diffusion objective | DARL · ImageNet, ViT-L | top-1 accuracy: 84.7 -> 84.9 | Helped |
| replace MSE with the diffusion objective | DARL · ImageNet, ViT-B | top-1 accuracy: 82.7 -> 81.9 | Hurt |
| replace MSE with the diffusion objective | DARL · ImageNet, ViT-H | top-1 accuracy: 85.5 -> 85.9 | Helped |
| replace 1D RoPE with 2D RoPE (ablation) | DARL with unsupervised pre-training · ImageNet | top-1 accuracy: 83.5 -> 84.5 | Helped |
| replace 1D RoPE with 2D RoPE (ablation) | DARL with unsupervised pre-training · ImageNet (resolution 384) | top-1 accuracy: 84.2 -> 85.1 | Helped |
| use a Transformer denoising patch decoder instead of an MLP decoder (ablation) | DARL with diffusion objective · ImageNet | top-1 accuracy: 82.8 -> 83.1 | Helped |
| fine-tune without causal attention masking instead of with causal masking (ablation) | DARL · ImageNet | top-1 accuracy: 81.7 -> 82.7 | Helped |
| predict the original image instead of noise (ablation) | DARL with diffusion objective · ImageNet | top-1 accuracy: 82.9 -> 83.6 | Helped |
| use normalized pixels instead of raw pixels as the reconstruction target (ablation) | DARL with MSE · ImageNet | top-1 accuracy: 82.7 -> 82.4 | Hurt |
| use nested raster ordering instead of raster ordering (ablation) | DARL · ImageNet | top-1 accuracy: 83.3 -> 83.4 | No clear effect |
| use round-robin ordering instead of raster ordering (ablation) | DARL · ImageNet | top-1 accuracy: 83.3 -> 82.7 | Hurt |
| encode image patches with dVAE instead of linear patch embedding (ablation) | DARL · ImageNet | top-1 accuracy: 83.6 -> 62.9 | Hurt |
Surprising findings
- Plain MSE loss alone gives strong representations; diffusion objective adds only a marginal gain at short training, and the optimal noise schedule concentrates on very high noise levels, unlike standard image diffusion schedules.
- Random patch ordering gives no final performance advantage over raster order, despite needing longer training.
Limitations
- Gap of about 1% to MAE on ImageNet attributed possibly to causal attention inductive bias; the ViTDet detection head may be suboptimal for generative pre-training.
- Denoising patch decoder is a single Transformer block, which may limit diffusion performance especially for large patch sizes.
- Scaling behavior is based on few model sizes, with insufficient data points for a statistically significant analysis.
Experimental setup
Datasets: ImageNet VTAB COCO
Benchmarks: ImageNet VTAB COCO
Metrics: top-1 accuracy AP mAP
Baselines: Supervised ViT MAE BeiT DINO MoCo v3 Image-GPT
Builds on: DDPM Vision Transformer XLNet Masked Autoencoders ViTDet
Closest papers by meaning
- Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (ICLR 2025 · similarity 0.90)
- Denoising Autoregressive Transformers for Scalable Text-to-Image Generation (ICLR 2025 · similarity 0.89)
- Deconstructing Denoising Diffusion Models for Self-Supervised Learning (ICLR 2025 · similarity 0.89)
- Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception (ICLR 2025 · similarity 0.88)
- Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think (NeurIPS 2025 · similarity 0.88)
- Boosting Generative Image Modeling via Joint Image-Feature Synthesis (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.