Research map / Image generation / Fast diffusion sampling
Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes score entropy, a loss that extends score matching to discrete spaces, to train SEDD discrete diffusion language models that match or beat GPT-2 perplexity and generate competitive text without annealing.
Problem
Diffusion models underperform autoregressive models on discrete data such as natural language, because prior discrete diffusion objectives (mean prediction, ratio matching, concrete score matching) are empirically weak or intractable.
Approach
Learn the concrete score (probability ratios p_t(y)/p_t(x)) with a score entropy loss derived from a Bregman divergence with F=-log, giving a tractable denoising variant (DWDSE) that bounds likelihood and plugs into discrete diffusion with absorbing or uniform transition matrices, plus a Tweedie tau-leaping sampler.
Key results
- SEDD Absorb achieves zero-shot perplexity beating GPT-2 small on WikiText2, PTB, WikiText103 (e.g., PTB 114.24 vs 138.43), and reduces perplexity 25-75% vs prior language diffusion models.
- Unconditional generative perplexity for SEDD is 6-8x better than un-annealed GPT-2 at 2048 steps and matches GPT-2 with 32x fewer network evaluations.
- Conditional infilling MAUVE of SEDD Standard is 0.957 vs GPT-2 with nucleus sampling at 0.955, with no annealing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an absorbing transition instead of GPT-2 Small | SEDD · WikiText2 | perplexity: 42.43 -> ≤41.84 | Helped |
| Use an absorbing transition instead of GPT-2 Small | SEDD · PTB | perplexity: 138.43 -> ≤114.24 | Helped |
| Use an absorbing transition instead of GPT-2 Small | SEDD · WikiText103 | perplexity: 41.60 -> ≤40.62 | Helped |
| Use an absorbing transition instead of GPT-2 Small | SEDD · LAMBADA | perplexity: 45.04 -> ≤50.92 | Hurt |
| Use an absorbing transition instead of GPT-2 Small | SEDD · One Billion Words | perplexity: 75.20 -> ≤79.29 | Hurt |
| Use SEDD Absorb instead of SEDD Uniform | SEDD · text8 | BPC: ≤ 1.47 -> ≤ 1.39 | Helped |
| Use SEDD Absorb instead of SEDD Uniform | SEDD · One Billion Words | perplexity: ≤ 40.25 -> ≤ 32.79 | Helped |
| Use SEDD Absorb instead of SEDD Uniform | SEDD · LAMBADA | perplexity: ≤65.40 -> ≤50.92 | Helped |
| Use SEDD Absorb instead of SEDD Uniform | SEDD · WikiText2 | perplexity: ≤50.27 -> ≤41.84 | Helped |
| Use SEDD Absorb instead of SEDD Uniform | SEDD · PTB | perplexity: ≤140.12 -> ≤114.24 | Helped |
| Use SEDD Absorb instead of SEDD Uniform | SEDD · WikiText103 | perplexity: ≤49.60 -> ≤40.62 | Helped |
| Use SEDD Standard instead of GPT-2 with nucleus sampling | conditional generation · Standard prompting | MAUVE: 0.955 -> 0.957 | Helped |
Surprising findings
- Concrete score matching with an l2 loss trains poorly, giving 3-4x higher likelihood loss in ablation.
- SEDD achieves competitive perplexity and generation quality without distribution annealing such as nucleus sampling or temperature scaling.
Limitations
- Gap remains with modern large language models; experiments only at GPT-2 small and medium scale.
- Generative quality relies on evaluation with GPT-2 large perplexity, which can be gamed by annealing; KV-cache-free sampling trades speed vs memory.
Experimental setup
Datasets: OpenWebText text8 One Billion Words WebText LAMBADA WikiText2 PTB WikiText103
Benchmarks: text8 One Billion Words LAMBADA WikiText2 PTB WikiText103 MAUVE
Metrics: bits per character perplexity generative perplexity MAUVE
Baselines: GPT-2 D3PM PLAID Diffusion-LM DiffusionBert SSD-LM Multinomial Diffusion Bayesian Flow Networks
Builds on: score matching D3PM concrete score matching DiT tau-leaping
Closest papers by meaning
- Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion (ICML 2025 · similarity 0.92)
- Efficient Perplexity Bound and Ratio Matching in Discrete Diffusion Language Models (ICLR 2025 · similarity 0.91)
- The Diffusion Duality (ICML 2025 · similarity 0.90)
- Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.90)
- Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling (ICLR 2025 · similarity 0.90)
- Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.