Research map / Image generation / Fast diffusion sampling

SemanticDLM+: Improving Diffusion LLMs through Bias-variance Trade-off in Transition Kernel Design

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Masked diffusion language model decoding

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes transition-kernel design in diffusion LMs via a bias-variance error decomposition, then proposes SemDLM+ (semantic kernel with global jump plus semantic-frequency sampling penalty) to fix semantic-basin low-diversity collapse, evaluated on LM1B and OpenWebText.

Problem

Diffusion language models are highly sensitive to the transition kernel; uniform diffusion scales well but is hard to fit, masking is easy to fit but lacks repair, and semantic diffusion underperforms due to a 'semantic basin' collapse into low-diversity text.

Approach

Decompose generation error into asymptotic bias, exposure bias and optimization variance, argue semantic kernels are a middle ground, and add a global transition plus a semantic-frequency logit penalty during sampling to escape semantic basins.

Key results

• On LM1B at 512 sampling steps, SemDLM+ reaches generation PPL 36.60 vs 70.74 for SEDD (Masking) and 75.98 for ReMDM.
• On OpenWebText at 1024 steps, SemDLM+ reaches generation PPL 34.37 vs 41.25 for ReMDM and 43.00 for MDLM.
• SemDLM+ PPL drops from 74.52 to 36.60 on LM1B as sampling steps increase, indicating benefit from extra reverse steps.

Surprising findings

• Extremely low generation perplexity with zero frequency penalty reflects a collapsed low-diversity regime rather than strong generation.
• Overly large gamma degrades generation quality because transitions get trapped in the semantic basin.

Limitations

• Semantic graph is token-level and cannot capture context-dependent meaning.
• Semantic penalty is an approximation that makes generation sensitive to hyperparameters.
• Generation PPL alone can be misleading; broader diversity evaluation needed.

Experimental setup

Datasets: LM1B OpenWebText

Benchmarks: LM1B OpenWebText

Metrics: test perplexity generation perplexity entropy

Baselines: D3PM (Uniform) MDLM ReMDM SEDD (uniform) SEDD (Masking) GIDD

Builds on: MDLM SEDD D3PM GIDD ReMDM

Closest papers by meaning

• From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs (NeurIPS 2026 · similarity 0.92)
• Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.92)
• Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.91)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.91)
• Theoretical Benefit and Limitation of Diffusion Language Model (NeurIPS 2025 · similarity 0.91)
• Denoising Time Matters: Diverse Generation in Diffusion Language Models (NeurIPS 2026 · similarity 0.90)