Research map / Image generation / Fast diffusion sampling
Simple Guidance Mechanisms for Discrete Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives discrete classifier-free and classifier-based guidance for discrete diffusion and introduces UDLM, a uniform-noise diffusion LM trained with a continuous-time ELBO that is more guidable and competitive in perplexity.
Problem
Controllable generation is hard for discrete data because continuous guidance relies on gradients with respect to discrete inputs, and discrete diffusion (especially uniform noise) lags autoregressive models in perplexity.
Approach
Adapts classifier-free and classifier-based guidance to discrete diffusion by tempering per-token distributions (no gradients w.r.t. discrete data; first-order Taylor approximation for the classifier), and pairs them with a uniform-noise diffusion model (UDLM/Mulan) whose tight continuous-time ELBO makes it closer to absorbing-state performance and repeatedly editable.
Key results
- On CIFAR10, D-CFG with MDLM reaches FID 15.56 and IS 9.02 versus 33.75 / 6.74 without guidance.
- On genomic sequences, UDLM with D-CFG at gamma=3 achieves F1 0.94 while AR's F1 degrades to 0.74 at gamma=3.
- UDLM (Mulan) achieves perplexity 3.15 on Species10 and 2.02 on QM9, competitive with or better than MDLM on small vocabularies.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the continuous-time NELBO (T→∞) instead of finite-step training (ablation) | UDLM on Amazon Polarity · Amazon Polarity | validation perplexity | Helped |
| Use UDLM instead of MDLM at T=128 (ablation) | CIFAR10 conditional generation with D-CFG · CIFAR10 (T=128) | FID: 64.09 -> 30.48 | Helped |
| Use UDLM instead of MDLM at T=1024 (ablation) | CIFAR10 conditional generation with D-CFG · CIFAR10 (T=1024) | FID: 27.94 -> 26.70 | Helped |
| Increase D-CFG strength from γ=1 to γ=3 (ablation) | UDLM genome generation · Species-specific genome generation | F1: 0.91$_{\pm\text{0.01}}$ -> \bf0.94$_{\pm\text{0.007}}$ | Helped |
| Increase D-CFG strength from γ=1 to γ=3 (ablation) | AR genome generation · Species-specific genome generation | F1: 0.87$_{\pm\text{0.011}}$ -> 0.74$_{\pm\text{0.022}}$ | Hurt |
| Increase D-CFG strength from γ=1 to γ=3 (ablation) | MDLM genome generation · Species-specific genome generation | F1: 0.88$_{\pm\text{0.011}}$ -> 0.78$_{\pm\text{0.007}}$ | Hurt |
| Add D-CFG | MDLM image generation · CIFAR10 | FID: 33.75 -> \bf15.56 | Helped |
| Add D-CFG | UDLM image generation · CIFAR10 | FID: 33.65 -> 23.21 | Helped |
| Use UDLM instead of MDLM | CIFAR10 image generation · CIFAR10 | FID: 33.75 -> 33.65 | No clear effect |
| Use UDLM instead of MDLM | QM9 language modeling · QM9 | perplexity: 2.12$_{\leq}$ -> 2.02$_{\leq}$ | Helped |
| Use UDLM instead of MDLM | CIFAR10 language modeling · CIFAR10 | perplexity: 9.14$_{\leq}$ -> 11.21$_{\leq}$ | Hurt |
| Use UDLM instead of MDLM | text8 language modeling · text8 | perplexity: 2.62$_{\leq}$ -> 2.71$_{\leq}$ | Hurt |
Surprising findings
- Uniform noise diffusion matches or beats absorbing-state diffusion on small-vocabulary datasets, contrary to prior belief.
- AR and MDLM control metrics degrade as the guidance strength increases, while UDLM keeps improving on genomic control.
Limitations
- Uniform-noise UDLM still trails absorbing-state diffusion on large-vocabulary NLP datasets (e.g., LM1B 31.28 vs 27.04 for MDLM).
- Classifier-based guidance requires an independence assumption across tokens conditioned on noisy latents and a first-order approximation for efficiency.
Experimental setup
Datasets: Species10 QM9 SMILES strings CIFAR10 text8 Amazon Review LM1B
Benchmarks: Species10 QM9 CIFAR10 text8 Amazon Review LM1B
Metrics: perplexity Disc. AUROC F1 validity novelty mean QED ring count FID IS
Baselines: AR (autoregressive) with D-CFG PPLM FUDGE NOS MDLM D3PM
Builds on: D3PM MDLM FUDGE NOS PPLM
Closest papers by meaning
- Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-based Decoding (NeurIPS 2025 · similarity 0.92)
- Unlocking Guidance for Discrete State-Space Diffusion and Flow Models (ICLR 2025 · similarity 0.91)
- Elucidating the design space of classifier-guided diffusion generation (ICLR 2024 · similarity 0.91)
- No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Models (ICLR 2025 · similarity 0.90)
- Token Perturbation Guidance for Diffusion Models (NeurIPS 2025 · similarity 0.89)
- TFG: Unified Training-Free Guidance for Diffusion Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.