Research map / Large language models / Reasoning and chain of thought

Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TraceRL, a trajectory-aware RL method for diffusion language models that optimizes over inference traces (with a diffusion-based value model), producing TraDo-4B/8B instruct models and the first long-CoT diffusion LM, TraDo-8B-Thinking.

Problem

Post-training of diffusion language models (DLMs) uses random-masking objectives that mismatch the block-wise, confidence-ordered inference trajectory actually used at sampling time, and no unified RL framework covers both full-attention and block-attention DLMs.

Approach

Optimize the policy over the model's own decoding trajectory (grouping every s neighboring denoising steps via a shrinkage parameter, and training block-attention slices in parallel), plus a diffusion-based value model that estimates step-wise values for token-level GAE advantages.

Key results

• TraDo-8B-Instruct reaches 78.5% on MATH500 (static) vs 74.0% for Qwen2.5-7B-Instruct, with 25.9% on LiveCodeBench-v2 vs 8.2% ... (relative gains reported as 6.1% over Qwen2.5-7B-Instruct and 51.3% over Llama3.1-8B-Instruct)
• TraDo-8B-Thinking achieves 87.4% on MATH500 and 35.5% on AIME2024, the first long-CoT diffusion LM.
• TraceRL yields 15.4% dynamic-sampling speedup on MATH500 for TraDo-4B-Instruct relative to SDAR-4B-Chat base, and a value model reduces training fluctuation.

Surprising findings

• Enlarging block size from 4 to 8 via TraceRL recovers the accuracy lost when inferring with block size 8 on the base model (MATH500 67.7 vs 60.2 at B=8 before RL).
• Preliminary study: semi-AR SFT with block size 16 on a full-attention model outperformed fully random masking even at equal compute.

Limitations

• Process-level rewards are proposed but not explored in experiments.
• Full-attention RL results are limited to a coding task with Dream-7B-Coder; less extensive coverage than block-attention models.

Experimental setup

Datasets: MATH training set PrimeIntellect verified coding problems OpenR1-MATH CodeContest

Benchmarks: MATH500 AIME2024 GSM8K LiveCodeBench-v2 LiveBench

Metrics: accuracy acceleration ratio average response length

Baselines: random masking RL coupled RL Qwen2.5-7B-Instruct Llama3.1-8B-Instruct LLaDA-8B-Instruct Dream-7B-Instruct SDAR-4B-Chat SDAR-8B-Chat

Builds on: GRPO PPO Block diffusion SDAR Dream-7B Fast-dLLM coupled RL

Closest papers by meaning

• d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation (ICML 2026 · similarity 0.91)
• Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models (NeurIPS 2025 · similarity 0.90)
• Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via Monotonic Entropy Descent with Reinforcement Learning (ICML 2026 · similarity 0.90)
• Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages (NeurIPS 2026 · similarity 0.89)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.89)