Research map / Image generation / Fast diffusion sampling
Think while You Generate: Discrete Diffusion with Planned Denoising
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DDPD, which splits discrete diffusion into a planner (predicts which positions are corrupted) and a denoiser (predicts values), sampled with an adaptive Gillespie-style sampler with time correction.
Problem
Discrete diffusion models lag autoregressive models in language and image token generation; denoiser-only designs cannot correct earlier mistakes and waste sampling steps.
Approach
Factor the reverse jump rate into a planner (probability each position is corrupted) and a denoiser (value prediction given corruption), train them separately from the ELBO, and sample with an adaptive Gillespie sampler that picks the most corrupted position each step and adjusts time from the planner's noise estimate.
Key results
- On text8, DDPD with separately trained planner and denoiser outperforms DFM with 2x parameters across sampling budgets (reported as superior quality-diversity trade-off).
- On ImageNet 256x256 token generation, DDPD reaches FID about 4.6-5.0 across 8-128 steps without logit annealing at 128 steps, vs 26.83 for standard mask diffusion at 128 steps.
- DDPD is the only tested method that performs consistently with or without heuristics or classifier-free guidance on ImageNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DDPD instead of standard mask diffusion | Image token generation · ImageNet 256×256, 32 steps, no logit annealing | FID: 29.12 -> 4.75 | Helped |
| Use DDPD instead of MaskGIT | Image token generation · ImageNet 256×256, 32 steps, no logit annealing | FID: 8.09 -> 4.75 | Helped |
| Add logit temperature 0.6 (ablation) | Standard mask diffusion · ImageNet 256×256 | FID: 29.12 -> 4.86 | Helped |
| Add logit temperature 0.6 (ablation) | MaskGIT · ImageNet 256×256, 32 steps | FID: 8.09 -> 11.93 | Hurt |
| Add logit temperature 0.6 (ablation) | DDPD · ImageNet 256×256, 32 steps | FID: 4.75 -> 4.91 | Hurt |
| Use linear logit temperature annealing from 1.0 to 0.0 (ablation) | Standard mask diffusion · ImageNet 256×256, 32 steps | FID: 29.12 -> 4.62 | Helped |
| Use linear logit temperature annealing from 1.0 to 0.0 (ablation) | MaskGIT · ImageNet 256×256, 32 steps | FID: 8.09 -> 9.9 | Helped |
| Use linear logit temperature annealing from 1.0 to 0.0 (ablation) | DDPD · ImageNet 256×256, 32 steps | FID: 4.75 -> 4.98 | Hurt |
| Use DDPD plan-and-denoise sampling instead of DFM | Text8 generation · text8 | NLL and sample entropy trade-off | Helped |
| Use DDPD-DFM-Uni single-network training instead of separately trained planner and denoiser (ablation) | Text8 generation · text8 | generation quality | Hurt |
| Train a separate planner and denoiser rather than a single model (ablation) | DDPD · text8, 20k training iterations | generation quality | Helped |
| Use a mask diffusion denoiser with a separately trained planner (ablation) | DDPD · OpenWebText | generative perplexity | Helped |
Surprising findings
- With only 20k training steps, separate planner+denoiser matches DFM trained for 750k steps on text8, while a single-network model suffers mode collapse.
- Spending compute on planning (2 NFE per step) is reported as more effective than doubling denoiser size or steps when cost is a factor.
Limitations
- Each sampling step costs two network evaluations (planner plus denoiser), an extra inference cost.
- Separate planner and denoiser training accumulates approximation errors, so single-network model is better when training is sufficient on simple tasks like text8.
Experimental setup
Datasets: text8 OpenWebText ImageNet
Benchmarks: text8 OpenWebText ImageNet 256x256
Metrics: negative log-likelihood under GPT-J-6B token entropy generative perplexity FID
Baselines: SEDD DFM DFM-Uni GPT-2 MaskGIT standard mask diffusion
Builds on: discrete flow matching SEDD MaskGIT tau-leaping
Closest papers by meaning
- Denoising Diffusion Bridge Models (ICLR 2024 · similarity 0.90)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.90)
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (ICLR 2025 · similarity 0.90)
- Constrained Discrete Diffusion (NeurIPS 2025 · similarity 0.89)
- Encoder-Decoder Diffusion Language Models for Efficient Training and Inference (NeurIPS 2025 · similarity 0.89)
- Compressed Image Generation with Denoising Diffusion Codebook Models (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.