Research map / Image generation / Fast diffusion sampling

Denoising Time Matters: Diverse Generation in Diffusion Language Models

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Masked diffusion language model decoding

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TAPS, a training-free diffusion-LM sampler that injects time-annealed, manifold-constrained perturbations into the prompt embedding during early denoising to boost semantic and lexical diversity while preserving quality and reasoning.

Problem

Diffusion language models can produce repetitive or conservative outputs under diversity-seeking prompts, and existing diversity controls (token-level sampling, prompt-based methods) do not exploit the denoising-time structure of diffusion inference.

Approach

Early denoising steps mainly determine high-level semantics, so perturbing the prompt conditioning embedding only during early steps (annealed to zero before refinement) and constraining it with distribution-aware rescaling, mixing with the clean embedding, and norm-preserving projection yields semantic branching without degrading later refinement.

Key results

• On NoveltyBench with LLaDA-8B-Instruct, TAPS raises Sent-BERT from 25.80 (base) to 36.04 and EAD from 50.22 to 63.35, while Div-BLEU rises from 48.44 to 66.26.
• On reasoning, TAPS improves Pass@5 over the base model on all four benchmarks (e.g., GPQA-Diamond 48.5 to 50.0; GSM8K 90.6 to 91.2) while keeping mean accuracy competitive.
• Computational overhead is +0.2% time per sample with no additional peak memory on a single-GPU-scale setup.

Surprising findings

• Diverse Prompt yields gains on LLaDA but degrades substantially on TraDo, and min-p degrades on TraDo, showing backbone-dependent baseline behavior.
• Token-level context masking gives comparable or higher lexical diversity gains than embedding perturbation but larger quality degradation.

Limitations

• Embedding-level perturbation hyperparameters (sigma, window, psi) are backbone-specific and tuned; TraDo is less tolerant of strong noise.
• EDT produced degenerate outputs under the authors' evaluation setting and was omitted from NoveltyBench comparison.
• Effect on RL rollout collection is untested; the theoretical bound relies on local Lipschitz assumptions and L_z<=1 for the simplified form.

Experimental setup

Datasets: WritingPrompts NoveltyBench Arena-Hard-Auto creative-writing GSM8K MATH500 GPQA-Diamond HumanEval+

Benchmarks: NoveltyBench WritingPrompts Arena-Hard-Auto creative-writing GSM8K MATH500 GPQA-Diamond HumanEval+

Metrics: IntraDistinct Div-BLEU Sent-BERT EAD mean accuracy Pass@5 majority-vote accuracy GPT-4o quality scores

Baselines: Top-k Top-p Min-p Diverse Prompt EDT

Builds on: CADS Diverse Prompt EDT min-p sampling

Closest papers by meaning

• Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.92)
• Scaling Inference Time Compute for Diffusion Models (CVPR 2025 · similarity 0.91)
• SemanticDLM+: Improving Diffusion LLMs through Bias-variance Trade-off in Transition Kernel Design (NeurIPS 2026 · similarity 0.90)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.90)
• Support Before Frequency in Discrete Diffusion (NeurIPS 2026 · similarity 0.90)
• MRO: Enhancing Reasoning in Diffusion Language Models via Multi-Reward Optimization (NeurIPS 2025 · similarity 0.90)