Research map / Image generation / Fast diffusion sampling
Discrete Copula Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Discrete Copula Diffusion (DCD), an inference-time method that combines a discrete diffusion model's univariate marginals with an autoregressive copula model's inter-variable dependencies via I-projection, enabling high-quality text generation with 8-32x fewer denoising steps.
Problem
Discrete diffusion models need hundreds to thousands of denoising steps because, at each step, they predict each variable independently and thus ignore inter-variable dependencies, leaving an irreducible term in the ELBO.
Approach
Combine the diffusion model's accurate univariate marginals with the dependency structure (copula) from a separately trained autoregressive model by I-projecting the copula distribution onto the diffusion marginals, which reduces to a convex problem approximated row-by-row; no fine-tuning of either model is needed.
Key results
- DCD with 4 denoising steps matches SEDD-Medium with 128 steps in unconditional generative perplexity, a 32x reduction in steps.
- On MAUVE text infilling across five prompt masks, DCD outperforms SEDD-M at equal step counts and GPT-2-S in all five tasks.
- On antibody infilling, DCD with 4 steps outperforms both NOS-D (64 steps) and the GPT copula model in sequence recovery.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine SEDD-M with GPT-2-S as a copula model | SEDD-M · WikiText-103 infilling, [0.1,0.2] & [0.5,0.7] | MAUVE: 0.201 -> 0.158 | Hurt |
| Combine SEDD-M with GPT-2-S as a copula model | SEDD-M · WikiText-103 infilling, [0.25,0.75] | MAUVE: 0.278 -> 0.314 | Helped |
| Combine SEDD-M with GPT-2-S as a copula model | SEDD-M · WikiText-103 infilling, [0.0,0.1] & [0.4,0.6] & [0.9,1.0] | MAUVE: 0.928 -> 0.983 | Helped |
| Combine SEDD-M with GPT-2-S as a copula model | SEDD-M · WikiText-103 infilling, [0.4,0.5] & [0.8,1.0] | MAUVE: 0.914 -> 0.981 | Helped |
| Combine SEDD-M with GPT-2-S as a copula model | SEDD-M · WikiText-103 infilling, [0.2,0.3] & [0.6,0.8] | MAUVE: 0.215 -> 0.403 | Helped |
| Use DCD with 4 denoising steps instead of SEDD-M with 128 steps | Unconditional text generation · WebText and OpenWebText | generative perplexity | No clear effect |
| Combine NOC-D with a GPT copula model | NOC-D · Antibody sequence infilling | sequence recovery rate | Helped |
Surprising findings
- Fully factorized diffusion denoising leaves an irreducible total-correlation term in the negative ELBO, so independent denoising caps few-step quality regardless of training.
Limitations
- Requires training or obtaining a separate copula model, which may need extra training for some applications.
- The I-projection is only approximated (row-wise update), so the guarantee holds only approximately in practice.
- Each denoising step is more expensive than the base diffusion model, so DCD may not give a notable wall-clock speedup.
Experimental setup
Datasets: OpenWebText WebText WikiText-103 Observed Antibody Space (OAS)
Benchmarks: OpenWebText generative perplexity WikiText-103 MAUVE infilling antibody sequence recovery rate
Metrics: generative perplexity MAUVE sequence recovery rate sampling time
Baselines: SEDD GPT-2-small MDLM SSD-LM NOS-D
Builds on: SEDD GPT-2 SSD-LM matrix scaling
Closest papers by meaning
- Fast Sampling via Discrete Non-Markov Diffusion Models with Predetermined Transition Time (NeurIPS 2024 · similarity 0.89)
- Constrained Discrete Diffusion (NeurIPS 2025 · similarity 0.88)
- Think while You Generate: Discrete Diffusion with Planned Denoising (ICLR 2025 · similarity 0.88)
- Continuous Diffusion Model for Language Modeling (NeurIPS 2025 · similarity 0.88)
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (ICLR 2025 · similarity 0.88)
- Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.