Research map / Image generation / Fast diffusion sampling
Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that the concrete score in absorbing discrete diffusion equals a time-independent clean-data conditional times a analytic scalar, and proposes RADD, a time-independent absorbing diffusion model with caching-based sampling and a unified loss view linking it to any-order autoregressive models.
Problem
Absorbing discrete diffusion models for language estimate the concrete score, whose scaling trick is not theoretically understood, and the time-conditioned network adds cost and prevents caching during sampling; the relation between diffusion objectives and AO-ARMs is also unclear.
Approach
The concrete score in absorbing diffusion factorizes into a time-dependent scalar times a time-independent conditional distribution of clean data given unmasked tokens; this justifies removing time conditioning (RADD) and caching network outputs between state changes, and shows the diffusion ELBO equals an AO-ARM objective.
Key results
- RADD-DSE small achieves 38.83 WikiText2 and 37.46 WikiText103 perplexity versus 41.84 and 40.62 for SEDD-Scale, with all four RADD losses beating SEDD on average across five datasets.
- RADD-AO medium reaches 28.51 on WikiText103 and 29.96 on WikiText2, the best among diffusion models at the medium scale in the table.
- Caching reduces expected NFEs as n(1-(1-1/n)^l), giving RADD with caching better perplexity and lower sampling time than SEDD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use scaled rather than unscaled SEDD (ablation) | SEDD absorbing diffusion · PTB (small) | perplexity: 130.49 -> 114.24 | Helped |
| Remove time-conditioning while using DSE loss (ablation) | SEDD-Scale / RADD-DSE · WikiText2 (small) | perplexity: 41.84 -> 38.83 | Helped |
| Train RADD with DSE instead of λ-DCE (ablation) | RADD · WikiText2 (small) | perplexity: 39.98 -> 38.83 | Helped |
| Train RADD with AO loss instead of DSE (ablation) | RADD · LAMBADA (medium) | perplexity: 42.30 -> 41.96 | Helped |
| Train RADD with λ-DCE instead of DSE (ablation) | RADD · PTB (medium) | perplexity: 75.16 -> 82.08 | Hurt |
| Train RADD with AO loss instead of DSE (ablation) | RADD · WikiText103 (medium) | perplexity: 28.03 -> 28.51 | Hurt |
| Train RADD with one of the four equivalent losses (ablation) | RADD · Five zero-shot language modeling datasets | perplexity | Mixed |
| Use RADD caching to reuse network outputs when the noisy sample is unchanged | RADD sampler · Language generation | number of function evaluations | Helped |
Surprising findings
- Removing time conditioning entirely does not hurt; RADD-DSE without time input outperforms time-conditioned SEDD-Scale under the same DSE loss.
Limitations
- Evaluated primarily at GPT-2 scale; scaling behavior untested.
- Can only generate full-length fixed-length outputs, unlike autoregressive models with variable length.
Experimental setup
Datasets: OpenWebText LAMBADA WikiText2 PTB WikiText103 One Billion Words
Benchmarks: LAMBADA WikiText2 PTB WikiText103 One Billion Words
Metrics: perplexity unigram entropy expected number of function evaluations running time
Baselines: SEDD-Scale SEDD-Unscale SEDD-Uniform D3PM PLAID GPT-2
Builds on: SEDD denoising score entropy any-order autoregressive models D3PM
Closest papers by meaning
- Support Before Frequency in Discrete Diffusion (NeurIPS 2026 · similarity 0.90)
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (ICLR 2025 · similarity 0.89)
- Absorb and Converge: Provable Convergence Guarantee for Absorbing Discrete Diffusion Models (NeurIPS 2025 · similarity 0.89)
- Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion (ICML 2025 · similarity 0.89)
- Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (ICML 2024 · similarity 0.89)
- O(d/T) Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.