Research map / Image generation / Fast diffusion sampling

Support Before Frequency in Discrete Diffusion

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Masked diffusion language model decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows via small-noise expansion of exact reverse kernels that discrete diffusion (uniform and absorbing) separates support (order of magnitude) from frequency (coefficient) information, and tests support-before-frequency on a FineWeb masked DLM and synthetic regular languages.

Problem

Understanding what structure of the data distribution discrete diffusion language models learn first from their denoising objectives; whether support information is exposed before frequency information.

Approach

The exact reverse kernel of uniform and absorbing diffusion expands in the small-noise limit into a scale sigma^{Delta d} (support signal) times a coefficient given by data probability ratios (frequency signal), so coarse multiplicative accuracy of scores suffices to recover support-improving edits, with absorbing diffusion concentrating leading mass on support-improving moves.

Key results

• On FineWeb, the frequency-matched support probe reaches 90% of peak gain at 116.3±5.7M tokens, versus 234.3±22.2M for top-1 mode and 247.4±7.5M for pairwise ranking.
• Thresholding the normalized uniform-diffusion reverse scores substantially improves sample validity on the synthetic regular language, while absorbing diffusion sees nearly no change versus ancestral sampling.
• Theory: uniform diffusion one-token edits split into three scales (sigma^-1, 1, sigma); absorbing diffusion keeps only support-improving edits at leading order.

Surprising findings

• Uniform-diffusion thresholded samplers can achieve a higher support-landing rate than absorbing diffusion, despite absorbing being closer to a projector within its reverse graph.
• Coarse multiplicative score accuracy (up to o(sigma^{-1/2})) can hold while total variation and KL remain far from the true kernel.

Limitations

• Analysis concerns the exact reverse process in the small-noise limit and does not provide a training theory for when models learn these terms.
• Hard support is brittle for natural language; rare valid strings may have tiny probability, motivating an epsilon-effective support.
• Absorbing diffusion cannot re-mask or revise incorrectly decoded tokens, limiting global support projection.

Experimental setup

Datasets: FineWeb synthetic regular language (bigram-like local-transition language)

Benchmarks: FineWeb held-out support and frequency probes synthetic regular-language validity

Metrics: pairwise support accuracy top-1 mode recovery accuracy pairwise ranking accuracy fraction of samples in support

Baselines: ancestral sampling

Builds on: D3PM SEDD SUBS parameterization

Closest papers by meaning

• Encoder-Decoder Diffusion Language Models for Efficient Training and Inference (NeurIPS 2025 · similarity 0.91)
• Denoising Time Matters: Diverse Generation in Diffusion Language Models (NeurIPS 2026 · similarity 0.90)
• Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.90)
• Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data (ICLR 2025 · similarity 0.90)
• A Theoretical Analysis of Why Masked Diffusion Models Mitigate the Reversal Curse (NeurIPS 2026 · similarity 0.90)
• Diffusion Beats Autoregressive in Data-Constrained Settings (NeurIPS 2025 · similarity 0.89)