Research map / Image generation / Fast diffusion sampling
The Diffusion Duality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows uniform-state discrete diffusion emerges from Gaussian diffusion via an argmax pushforward, and uses this to build Duo, with a low-variance curriculum (2x faster training) and discrete consistency distillation (two orders of magnitude fewer sampling steps).
Problem
Uniform-state discrete diffusion language models lag behind autoregressive and masked diffusion models in likelihood and cannot easily use few-step distillation techniques developed for Gaussian diffusion.
Approach
Uniform-state diffusion emerges as the argmax pushforward of a Gaussian diffusion (the "diffusion duality"), which lets Gaussian-space tools be transferred: a tempered-softmax curriculum for lower-variance training and discrete consistency distillation along deterministic Gaussian PF-ODE trajectories projected by argmax.
Key results
- Duo reaches 33.7 PPL on LM1B and 25.2 on OWT, beating UDLM (36.7 and 27.4) and nearly closing the gap with MDLM (31.8, 23.2).
- Curriculum learning reaches PPL 35.2 after 10K fine-tuning steps, nearly 1.5 points better than UDLM at 1M steps, a 2x training speedup.
- Discrete consistency distillation reduces sampling from 1024 to 16 NFEs (ancestral) or 8 NFEs (Greedy-Tail) while matching base generative perplexity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add curriculum learning (ablation) | Duo · OWT zero-shot | PPL: 35.0 -> 33.7 | Helped |
| add Rao-Blackwellized training loss (ablation) | Duo without curriculum learning · OWT zero-shot | PPL: 36.7 -> 35.0 | Helped |
| use curriculum learning | Duo training on LM1B · LM1B | training speed: 2× | Helped |
| use curriculum learning | Duo training · LM1B | gradient variance: an order of magnitude | Helped |
| distill with DCD and use ancestral sampling | Duo · OWT text generation | sampling steps: 1024 to 16 | Helped |
| use Greedy-Tail sampling with distilled Duo | Duo · OWT text generation | sampling steps: 1024 to 8 | Helped |
| use Greedy-Tail instead of ancestral sampling | distilled Duo · OWT text generation | sample entropy | Hurt |
| use denoising-model weights rather than EMA weights as the DCD teacher (ablation) | Duo distillation · OWT text generation | Gen PPL | Helped |
| use curriculum with fixed temperature τ=0.1 (ablation) | Duo training · LM1B | training loss | Hurt |
Surprising findings
- Duo surpasses an autoregressive transformer on 3 of 7 zero-shot perplexity datasets despite being a diffusion model.
- Duo without curriculum learning still outperforms prior uniform-state and Gaussian diffusion models, because of the Rao-Blackwellized objective alone.
Limitations
- Curriculum learning has hyperparameters (tau and curriculum duration) that may need manual tuning in other domains.
- Gaussian samplers underperform discrete samplers in likelihood, complicating guidance use cases.
Experimental setup
Datasets: LM1B OpenWebText
Benchmarks: LM1B OpenWebText PTB WikiText Lambada AG News Pubmed Arxiv
Metrics: perplexity generative perplexity entropy NFEs
Baselines: SEDD Uniform UDLM Plaid MDLM SEDD Absorb D3PM Absorb Transformer autoregressive SDTT
Builds on: Uniform-state discrete diffusion (SEDD Uniform / UDLM) consistency distillation Gaussian diffusion / PF-ODE SDTT
Closest papers by meaning
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.91)
- Fast Sampling via Discrete Non-Markov Diffusion Models with Predetermined Transition Time (NeurIPS 2024 · similarity 0.91)
- Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.91)
- Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.91)
- Energy-Based Diffusion Language Models for Text Generation (ICLR 2025 · similarity 0.91)
- Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.