Research map / Large language models / Reasoning and chain of thought

Anchored Diffusion Language Model

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ADLM, a two-stage diffusion language model whose anchor network predicts low-frequency "important" tokens before a denoiser predicts masked tokens, trained with an anchored NELBO; it narrows the gap to AR models and is the first DLM to beat AR on MAUVE.

Problem

Masked diffusion language models lag autoregressive models in likelihood (perplexity) and generated text quality, partly because important tokens masked early in the forward process lose context needed for reconstruction.

Approach

Split each denoising step into an anchor network that predicts distributions over important (low-frequency) tokens and a denoiser conditioned on those anchor predictions, trained jointly via an Anchored Negative ELBO; anchoring is also applied to GPT-2 SFT as Anchored Chain-of-Thought.

Key results

• On LM1B, ADLM reaches 26.40 PPL at 33B tokens vs 27.07 for retrained MDLM; on OWT at 524B tokens ADLM reaches 20.14 PPL vs 22.98 for MDLM.
• With the remasking sampler at T=2048 on OWT, ADLM achieves MAUVE 0.788, above the AR baseline's 0.760.
• On GSM8K, ACoT attains 45.2% accuracy, above standard CoT (42.9%) and Coconut (34.1%).

Measured effects reported in the paper

Surprising findings

• ADLM reports MAUVE above the AR baseline, the first DLM to do so under the paper's setup.
• Applying the anchored objective to MDLM alone (gamma=0, multi-stage) lowers OWT PPL from 23.17 to 21.79, suggesting gains are not just extra capacity.

Limitations

• Token importance is defined by a simple low-frequency criterion (threshold tau=5), which may not generalize across tasks.
• Anchor selection for reasoning tasks uses task-specific heuristics (pruning operators, conjunctions) rather than a general criterion.

Experimental setup

Datasets: LM1B OpenWebText GSM8K ProntoQA ProsQA Lambada PTB Wikitext AG News PubMed ArXiv

Benchmarks: LM1B OpenWebText Lambada PTB Wikitext AG News PubMed ArXiv LM1B zero-shot GSM8K ProntoQA ProsQA

Metrics: perplexity GPT-2 Large perplexity entropy MAUVE accuracy

Baselines: AR Transformer SEDD MDLM BD3LM ReMDM GIDD DFM Forward-Backward CoT iCoT Coconut Pause Token

Builds on: MDLM SEDD ReMDM D3PM Coconut

Closest papers by meaning

• Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.92)
• Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.91)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.90)
• Diffusion Beats Autoregressive in Data-Constrained Settings (NeurIPS 2025 · similarity 0.90)
• From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs (NeurIPS 2026 · similarity 0.90)
• Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.90)