Research map / Image generation / Fast diffusion sampling
Diffusion Beats Autoregressive in Data-Constrained Settings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows masked diffusion language models outperform autoregressive models when training repeats limited data over many epochs, and derives data-constrained scaling laws and a critical-compute threshold.
Problem
Whether autoregressive or masked diffusion language models are more efficient when unique training data is limited and must be reused over many epochs, since prior diffusion-vs-AR comparisons only used single-epoch training.
Approach
Decouples model scaling from data reuse by training ~200 AR and masked diffusion models on fixed unique-token budgets (25M-500M) for many epochs, fitting data-constrained scaling laws (Muennighoff-style) for each family, and attributing diffusion's advantage to exposure to diverse token orderings from random masking, which AR can partially replicate with permutation augmentation.
Key results
- Best diffusion model reaches validation loss 3.55 vs 3.71 for best AR at 100M unique tokens, with AR best at ~50 epochs and diffusion best at ~500 epochs.
- Fitted R_D* is about 513 for diffusion vs 32 for AR; diffusion repeated data stays nearly as useful as fresh data up to ~100 epochs, versus ~4 for AR.
- Diffusion outperforms AR on 8 of 9 downstream tasks at 100M unique tokens (e.g., SciQ 68.67 vs 58.05), and critical compute scales as U^2.174.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with repeated data for extended epochs rather than stopping at single-epoch compute-optimal training | Diffusion versus AR · 100M unique tokens, validation | validation loss: 10.65 vs. 7.07; 3.55 vs. 3.71 | Helped |
| Use diffusion instead of AR | 100M unique-token language models · ARC-Easy, BoolQ, COPA, HellaSwag, RACE, WinoGrande XL, SciQ, Lambada | accuracy (%) | Helped |
| Use diffusion instead of AR | 100M unique-token language models · PiQA | accuracy (%): 60.94 -> 60.72; 61.64 -> 60.72 | Hurt |
| Use diffusion instead of AR | 500M unique-token language models · Downstream benchmarks | accuracy (%) | Mixed |
| Add attention dropout (25–75%) to AR training (ablation) | AR, 140M parameters, 50 epochs · Validation | validation loss | No clear effect |
| Add token masking to AR training (ablation) | AR, 140M parameters, 50 epochs · Validation | validation loss | No clear effect |
| Add randomized token orderings during AR training (ablation) | AR, 278M parameters, 100 epochs · Validation | validation loss | Helped |
Surprising findings
- Adding attention dropout or token masking to AR training did not reduce overfitting or close the gap with diffusion.
- AR trained with 16 random token orderings approaches diffusion's 100-epoch validation loss, suggesting ordering diversity drives the advantage.
Limitations
- Scaling laws fit over a limited range of unique data sizes (25M-500M) and model sizes (up to 2.5B).
- AR is more compute-efficient at low compute; diffusion only wins beyond a critical compute threshold.
- Hybrid AR-diffusion models suggested but not evaluated in data-efficiency lens.
Experimental setup
Datasets: C4 English OSCAR TinyStories WikiText IndustryCorpus2 EN Sub
Benchmarks: ARC-Easy BoolQ COPA HellaSwag PiQA RACE WinoGrande XL SciQ Lambada OSCAR TinyStories WikiText
Metrics: validation loss (NLL) accuracy R^2 of scaling-law fit NLL
Baselines: autoregressive transformer (GPT-2 style) AR with attention dropout AR with token masking AR with permutation orderings
Builds on: Chinchilla scaling laws Muennighoff data-constrained scaling laws masked diffusion language models RoPE
Closest papers by meaning
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.94)
- Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.93)
- Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.92)
- From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs (NeurIPS 2026 · similarity 0.92)
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (ICLR 2025 · similarity 0.92)
- Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.