Research map / Image generation / Fast diffusion sampling
On the Error Correcting Effects of Stochasticity in Discrete Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how stochasticity in discrete diffusion CTMC transitions trades sampling speed for error correction, and proposes training-free DCRS (churn and restart sampling) that alternates forward/reverse steps to improve low-NFE image generation.
Problem
Discrete diffusion sampling must balance speed and sample quality, and the role of transition stochasticity in propagating or correcting model approximation errors is poorly understood.
Approach
Redundant mutual transitions between state pairs exchange mass without changing marginals; an information-theoretic (SDPI/KL contraction) analysis shows these transitions contract accumulated errors. DCRS injects controlled stochasticity by restarting via closed-form forward corruption and churning with small forward-reverse perturbations, requiring no extra network evaluations for restarts.
Key results
- On CIFAR10 and CelebA, DCRS achieves comparable FID with significantly fewer steps (20-30 NFE regime); the paper reports up to 10x reduction in sampling steps on image datasets.
- On language benchmarks, reducing stochasticity has limited impact on generative perplexity for UDLM and MDLM; gains depend on corruption process and nucleus sampling.
- In the 1D controlled experiment, DPF converges faster with exact scores, while tau-leaping achieves lower error under perturbed scores.
Surprising findings
- Error-correcting benefits of stochasticity in masked diffusion emerge only in conjunction with temperature scaling; stochasticity alone did not help there.
- On language tasks, the error-correcting effect of ReMDM stochasticity disappears without nucleus sampling, and deterministic and stochastic samplers perform comparably for UDLM.
Limitations
- Need to tune the level and schedule of stochasticity and restart hyperparameters; hyperparameter selection from continuous diffusion does not transfer well.
- Gains on language benchmarks are nuanced and not consistent; higher-order solvers outside the restart window can compound discretization errors.
- Theory is mainly one-dimensional; extension to high-dimensional settings is outlined rather than fully established.
Experimental setup
Datasets: CIFAR10 CelebA LM1B OpenWebText 2D 8-mode mixture of Gaussians (projected to 128 bits)
Benchmarks: CIFAR10 CelebA OpenWebText LM1B 2D mixture of Gaussians
Metrics: FID generative perplexity MAUVE entropy Wasserstein-1 distance KL divergence
Baselines: tau-leaping DPF sampling ReMDM-Cap ReMDM-Loop MDLM UDLM
Builds on: discrete probability flow DDIM ReMDM tau-leaping MDLM
Closest papers by meaning
- Cross-fluctuation phase transitions reveal sampling dynamics in diffusion models (NeurIPS 2025 · similarity 0.89)
- Jump Your Steps: Optimizing Sampling Schedule of Discrete Diffusion Models (ICLR 2025 · similarity 0.89)
- Towards Non-Asymptotic Convergence for Diffusion-Based Generative Models (ICLR 2024 · similarity 0.88)
- Systematic Hazard Sampling: Minimal-Variance Inference for Discrete Diffusion and Flow Models (NeurIPS 2026 · similarity 0.88)
- O(d/T) Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions (ICLR 2025 · similarity 0.88)
- Fast Sampling via Discrete Non-Markov Diffusion Models with Predetermined Transition Time (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.