Research map / Large language models / Reasoning and chain of thought

d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adapts pretrained masked diffusion LLM LLaDA-8B-Instruct into a reasoning model via masked SFT on s1K followed by diffu-GRPO, a policy-gradient RL method using one-step masked-prompt log-probability estimation.

Problem

Reinforcement learning post-training has advanced reasoning in autoregressive LLMs, but it is unclear how to apply policy gradient RL to masked diffusion LLMs, whose sequence log-probabilities do not factorize autoregressively.

Approach

Estimates per-token log-probabilities of a masked dLLM with a single forward pass on a randomly masked prompt, uses a mean-field sequence log-probability, and plugs these into a GRPO-style objective (diffu-GRPO); random prompt masking acts as regularization that allows more gradient updates per batch.

Key results

• d1-LLaDA reaches 82.1 on GSM8K at seq len 512 and 40.2 on MATH500 at 512, versus 78.2 and 36.2 for LLaDA-8B-Instruct.
• On Countdown (seq len 512), d1-LLaDA scores 42.2 vs 16.0 for LLaDA-8B-Instruct; on Sudoku (seq len 128) 22.1 vs 11.7.
• diffu-GRPO outperforms LLaDA-8B-Instruct and SFT alone in all 12 math and planning setups, and d1-LLaDA beats diffu-GRPO-only in 11 of 12.

Measured effects reported in the paper

Surprising findings

• Random prompt masking during log-probability estimation lets the number of gradient updates per batch scale to 12 or 24 with stable learning, whereas fixed masking underperforms.
• SFT on s1K did not help coding; the authors suggest s1K lacks code datapoints.

Limitations

• Fixed-length generation in LLaDA constrains RL training to a predefined sequence length (256), limiting discovery of variable-length reasoning.
• Online RL generation is slow, so longer generation lengths during RL are currently infeasible.
• Gains on Sudoku at seq len 512 are mixed; diffu-GRPO alone scores higher than d1 there.

Experimental setup

Datasets: s1K GSM8K MATH KodCode-Light-RL-10K TinyZero Countdown 4x4 Sudoku dataset

Benchmarks: GSM8K MATH500 Countdown Sudoku HumanEval MBPP

Metrics: accuracy correctness reward (Sudoku cell fraction)

Baselines: LLaDA-8B-Instruct LLaDA+SFT LLaDA+diffu-GRPO

Builds on: GRPO LLaDA s1K PPO Monte Carlo log-likelihood estimation

Closest papers by meaning

• Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via Monotonic Entropy Descent with Reinforcement Learning (ICML 2026 · similarity 0.93)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.92)
• d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation (ICML 2026 · similarity 0.92)
• On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.91)
• Lavida-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Models (ICML 2026 · similarity 0.91)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.91)