Research map / Image generation / Fast diffusion sampling
Efficient Perplexity Bound and Ratio Matching in Discrete Diffusion Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves three KL/cross-entropy theorems for discrete diffusion (CTMC), derives a tighter perplexity bound J2, and proposes CEDD (denoising cross-entropy instead of score entropy) plus a roulette transition matrix with analytic exponential.
Problem
Evaluating discrete diffusion language models requires a perplexity bound, and existing ratio-matching training (SEDD, score entropy) is costly and not clearly optimal for categorical data.
Approach
Derives discrete analogues of continuous-diffusion KL theorems yielding a cheaper cross-entropy upper bound J2; trains with a weighted denoising cross-entropy over per-position token posteriors (CEDD) instead of ratio matching, and introduces a roulette transition-rate matrix interpolating absorb and uniform diffusion with a closed-form matrix exponential.
Key results
- CEDD* absorb (L=128) reaches GenPerp 143.86 vs SEDD 172.35, roughly 10% lower generative perplexity.
- CEDD/CEDD* train about 15% faster per iteration than SEDD due to the simplified loss.
- The J2 bound is slightly tighter than J1 in all reported cases, e.g. CEDD* absorb L=1024 LAMBADA 52.10 (J2) vs 52.70 (J1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace score-entropy training with CEDD* | Absorb diffusion · OpenWebText, L=128 | generative perplexity: 166.35 -> 143.86 | Helped |
| Replace score-entropy training with CEDD* | Roulette diffusion · OpenWebText, L=128 | generative perplexity: 172.93 -> 158.56 | Helped |
| Replace score-entropy training with CEDD | Uniform diffusion · OpenWebText, L=128 | generative perplexity: 163.88 -> 161.84 | Helped |
| Use CEDD* instead of CEDD (ablation) | Absorb diffusion · OpenWebText, L=128 | generative perplexity: 148.21 -> 143.86 | Helped |
| Use CEDD* instead of CEDD (ablation) | Roulette diffusion · OpenWebText, L=128 | generative perplexity: 167.67 -> 158.56 | Helped |
| Use CEDD* instead of CEDD (ablation) | Uniform diffusion · OpenWebText, L=128 | generative perplexity: 161.84 -> 175.42 | Hurt |
| Replace CEDD* with CEDDT (ablation) | Absorb diffusion · OpenWebText, L=1024 | generative perplexity: 101.83 -> 108.88 | Hurt |
| Use DFM with schedule k_t=t instead of CEDD* | Language modeling · OpenWebText, L=128 | generative perplexity: 143.86 vs 145.48 | Mixed |
| Use DFM schedule k_t=t^2 instead of k_t=t (ablation) | Discrete flow matching · OpenWebText, L=128 | generative perplexity: 145.48 -> 152.70 | Hurt |
| Use CEDD* Roulette instead of SEDD Roulette | Spelling correction · PAP | correction accuracy: 85.1 -> 89.7 | Helped |
| Use CEDD* Roulette instead of SEDD Roulette | Spelling correction · CAP | correction accuracy: 85.8 -> 90.3 | Helped |
| Use CEDD* instead of SEDDs | Absorb diffusion · LAMBADA, L=1024 | perplexity bound: 52.18 -> 52.70 | Hurt |
Surprising findings
- Their CEDD* scaling outperforms the CEDDT scaling used in the SOTA score-entropy setup, despite the latter's theoretical support for score entropy.
- CEDD uniform performs worse than SEDD on PTB perplexity (279.65 vs 252.81), and the uniform variant is not uniformly better.
Limitations
- Heuristic rescaling of sigma and time-dependent loss weights are needed; noise schedules were not systematically explored.
- Roulette p_m was chosen from only four values; the best value found (0.95) is likely not optimal, with p_m=1 probably better.
- Results are on small models trained for 400k steps, with limited scale and only one dataset for training.
Experimental setup
Datasets: OpenWebText War and Peace 1BW LAMBADA PTB WikiText2 WikiText103 Crime and Punishment Pride and Prejudice
Benchmarks: LAMBADA WikiText2 PTB WikiText103 1BW OpenWebText generative perplexity spelling correction (Crime and Punishment, Pride and Prejudice)
Metrics: perplexity bound J1 perplexity bound J2 generative perplexity correction accuracy
Baselines: SEDD SEDD scaled CEDD T DFM GPT-2 D3PM PLAID
Builds on: SEDD score entropy D3PM tau-leaping
Closest papers by meaning
- Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (ICML 2024 · similarity 0.91)
- Continuous Diffusion Model for Language Modeling (NeurIPS 2025 · similarity 0.91)
- LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling (NeurIPS 2026 · similarity 0.90)
- Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion (ICML 2025 · similarity 0.89)
- Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling (ICLR 2025 · similarity 0.89)
- Discrete Diffusion Models: Novel Analysis and New Sampler Guarantees (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.