Research map / Image generation / Fast diffusion sampling
Scaling Diffusion Language Models via Adaptation from Autoregressive Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts open autoregressive LMs (GPT2 127M/355M and LLaMA2 7B) into masked discrete diffusion language models (DiffuGPT, DiffuLLaMA) via continual pre-training with attention mask annealing and a logit shift, using under 200B tokens.
Problem
Diffusion language models lag far behind autoregressive LLMs in scale and are costly to train from scratch, and their evaluation has relied mainly on perplexity, making fair comparison to AR models difficult.
Approach
Unify the AR cross-entropy objective with the absorbing discrete diffusion ELBO, then convert pretrained AR models into diffusion LMs by annealing the causal mask to full bidirectional attention, keeping the AR logit shift, and dropping time embeddings, followed by continual pre-training.
Key results
- DiffuLLaMA (7B) reaches 63.1% on GSM8K after LoRA finetuning, versus 58.6% for LLaMA2-7B, and 15.5 pass@1 on HumanEval infilling, versus 1.7 for LLaMA2 (prefix-only).
- DiffuGPT-M (355M) reaches 61.8% on GSM8K finetuning, versus 45.6% for GPT2-M, and outperforms GPT2-M on most commonsense tasks (e.g., HellaSwag 37.2 vs 38.3 is a slight drop, but Winogrande 52.6 vs 50.7).
- DiffuGPT-M achieves 17.0 ROUGE-L on ROCStories infilling, versus 8.2 for GPT2-M prefix-only, and state-of-the-art among diffusion LMs on the reported tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt GPT2-S to DiffuGPT-S | GPT2-S · HellaSwag | accuracy: 29.9 -> 33.4 | Helped |
| Adapt GPT2-S to DiffuGPT-S | GPT2-S · GSM8K | accuracy: 44.8 -> 50.2 | Helped |
| Adapt GPT2-M to DiffuGPT-M | GPT2-M · Winogrande | accuracy: 50.7 -> 52.6 | Helped |
| Adapt GPT2-M to DiffuGPT-M | GPT2-M · PIQA | accuracy: 67.4 -> 59.6 | Hurt |
| Adapt LLaMA2 to DiffuLLaMA | LLaMA2 · GSM8K | accuracy: 58.6 -> 63.1 | Helped |
| Adapt LLaMA2 to DiffuLLaMA | LLaMA2 · TriviaQA | accuracy: 45.4 -> 18.5 | Hurt |
| Add the shift operation during discrete diffusion adaptation (ablation) | GPT2-M + DD · GSM8K-symbolic | accuracy: 34.5 -> 49.7 | Helped |
| Add attention mask annealing during discrete diffusion adaptation (ablation) | GPT2-M + DD · GSM8K-symbolic | accuracy: 47.2 -> 49.7 | Helped |
| Replace discrete diffusion loss with continuous diffusion loss when adapting GPT2-M (ablation) | GPT2-M · GSM8K-symbolic | accuracy: 49.7 -> 20.2 | Hurt |
| Fine-tune GPT2-M with discrete diffusion loss instead of autoregressive loss (ablation) | GPT2-M · GSM8K-symbolic | accuracy: 45.6 -> 49.7 | Helped |
| Use DiffuGPT-M rather than GPT2-M as the discrete diffusion fine-tuning base (ablation) | GSM8K-symbolic fine-tuning · GSM8K-symbolic | accuracy: 49.7 -> 61.8 | Helped |
| Continue pre-training GPT2-M on the DiffuGPT corpus (ablation) | GPT2-M · HellaSwag | accuracy: 38.3 -> 36.7 | Hurt |
Surprising findings
- Continual pre-training of GPT2 on the same corpus gave no zero-shot improvement, while the diffusion adaptation did improve many tasks.
- Chain-of-thought prompting hurt DiffuLLaMA few-shot performance (MAWPS 31.3 FS vs 28.7 CoT; SATMath 23.6 vs 9.5), attributed to lack of instruction tuning.
Limitations
- DiffuLLaMA still falls well short of LLaMA2 on TriviaQA and PIQA, likely due to limited adaptation tokens and loss of general knowledge.
- Adapted DLMs show weak zero-shot in-context learning and CoT performance without instruction tuning.
- Training data (30B-65B tokens) is small relative to original pretraining, and AR baselines are not fine-tuned for infilling, which may make comparisons unfair.
Experimental setup
Datasets: FineWeb SlimPajama StarCoder GSM8K-symbolic TriviaQA LAMBADA HellaSwag Winogrande SIQA PIQA ROCStories HumanEval single-line infilling
Benchmarks: TriviaQA LAMBADA HellaSwag Winogrande SIQA PIQA GSM8K ROCStories HumanEval infilling MAWPS SAT-math
Metrics: accuracy exact match ROUGE pass@1 perplexity (GPT2-large) distinct 2-gram diversity
Baselines: SEDD Plaid 1B MD4 GPT2 LLaMA2
Builds on: SEDD absorbing discrete diffusion GPT-2 LLaMA 2 TinyLlama
Closest papers by meaning
- Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.93)
- Energy-Based Diffusion Language Models for Text Generation (ICLR 2025 · similarity 0.93)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.93)
- Diffusion Beats Autoregressive in Data-Constrained Settings (NeurIPS 2025 · similarity 0.93)
- From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs (NeurIPS 2026 · similarity 0.93)
- Continuous Latent Diffusion Language Model (NeurIPS 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.