Research map / Image generation / Fast diffusion sampling

Information-Theoretic Discrete Diffusion

NeurIPS 2025 · Image generation › Fast diffusion sampling · approach: Diffusion theory and learned samplers

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives I-MDSE and I-MDCE identities linking mutual information decay in discrete (masked) diffusion to minimum DSE/DCE losses, giving exact log-likelihood decompositions plus a time-free, coupled Monte Carlo estimator for likelihoods and likelihood ratios.

Problem

Discrete diffusion losses (DSE, DCE) have been viewed only as variational upper bounds on negative log-likelihood, and discrete diffusion lacks an information-theoretic counterpart to the I-MMSE identity used in Gaussian diffusion.

Approach

Establishes a discrete I-MMSE analogue: the derivative of mutual information (or pointwise KL) along the CTMC forward process equals the negative minimum DSE loss, and for masked absorbing diffusion equals the minimum DCE loss with a 1/λ weight, yielding exact (not bounded) NLL decompositions; this leads to a time-free estimator via random unmasking subsets with harmonic-number weights and a coupl

Key results

• Time-free conditional likelihood estimator reduces MC variance versus time-integral estimator; e.g., HellaSwag at 128 MC samples: 70.97 (time-integral) vs 11.57 (time-free).
• Coupled likelihood-ratio estimator on BeaverTails has far lower variance than decoupled: 8897.08 vs 62469.41 at 5 MC samples.
• Toy experiments show estimated unconditional and conditional NLLs closely match ground truth on DNA data.

Measured effects reported in the paper

Limitations

• Framework currently applies only to masked (absorbing) diffusion via I-MDCE; full I-MDSE setting left for future work.
• Likelihood recovery may expose sensitive information, requiring caution in privacy-critical deployment.

Experimental setup

Datasets: synthetic DNA sequences text8 HellaSwag ARC-hard PIQA BeaverTails WikiText-2 pretrain_zh

Benchmarks: HellaSwag ARC-hard PIQA BeaverTails

Metrics: Monte Carlo variance NLL agreement with ground truth NLL histograms for OOD separation

Baselines: time-integral DCE estimator decoupled likelihood ratio estimator LLaDA

Builds on: I-MMSE identity denoising score entropy RADD LLaDA

Closest papers by meaning

• Conservation Laws for Diffusion Models (NeurIPS 2026 · similarity 0.92)
• O(d/T) Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions (ICLR 2025 · similarity 0.90)
• Assessing the quality of denoising diffusion models in Wasserstein distance: noisy score and optimal bounds (NeurIPS 2025 · similarity 0.88)
• Adapting to Unknown Low-Dimensional Structures in Score-Based Diffusion Models (NeurIPS 2024 · similarity 0.88)
• Generalization through variance: how noise shapes inductive biases in diffusion models (ICLR 2025 · similarity 0.88)
• Convergence of Score-Based Discrete Diffusion Models: A Discrete-Time Analysis (ICLR 2025 · similarity 0.87)