Research map / Reinforcement learning / RL from feedback
Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DCoLT, outcome-reward RL over the entire reverse diffusion trajectory of diffusion language models, instantiated on SEDD (400M) and LLaDA 8B (as LLaDOU with a Plackett-Luce unmasking policy module) for math and code.
Problem
Reasoning in autoregressive LLMs is forced into a single causal left-to-right direction, while diffusion language models' intermediate denoising steps could support non-linear reasoning; prior DLM RL methods only optimize the final step.
Approach
Treat every intermediate reverse-diffusion step as a latent thinking action and optimize the whole trajectory with GRPO using only final-answer correctness as reward; for masked DLMs, add a Unmask Policy Module that ranks tokens with a Plackett-Luce model so the unmasking order is learned as part of the policy.
Key results
- LLaDOU (LLaDA + DCoLT) 8B reaches 88.1% on GSM8K, 44.6% on MATH, 51.6% on MBPP and 59.1% on HumanEval in the zero-shot setting, versus 78.3%, 38.9%, 40.2%, 39.6% for LLaDA 8B.
- SEDD 400M + DCoLT reaches 96.2% on Sudoku 4x4 and 57.0% on GSM8K-Aug, above SEDD + DoT (79.4%, 53.5%).
- On LLaDA GSM8K ablation, freezing LLaDA and training only the UPM raises accuracy from 47.27% to 69.24%; joint training reaches 81.06%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply outcome-based DCoLT reinforcement learning | SEDD 400M · Sudoku 4×4 | accuracy: 79.4 -> 96.2 | Helped |
| apply outcome-based DCoLT reinforcement learning | SEDD 400M · GSM8K-Aug | accuracy: 53.5 -> 57.0 | Helped |
| apply DCoLT reinforcement learning | LLaDA 8B · GSM8K | accuracy: 78.3 -> 88.1 | Helped |
| apply DCoLT reinforcement learning | LLaDA 8B · MATH | accuracy: 38.9 -> 44.6 | Helped |
| apply DCoLT reinforcement learning | LLaDA 8B · HumanEval | pass@1: 39.6 -> 59.1 | Helped |
| apply DCoLT reinforcement learning | LLaDA 8B · MBPP | pass@1: 40.2 -> 51.6 | Helped |
| train the UPM with AdaLN while freezing LLaDA (ablation) | LLaDA 8B with DCoLT · GSM8K | accuracy: 47.27 -> 69.24 | Helped |
| train the LLaDA part with RL alongside the UPM (ablation) | LLaDA 8B with DCoLT · GSM8K | accuracy: 69.24 -> 81.06 | Helped |
| add AdaLN to the UPM (ablation) | LLaDA 8B with DCoLT · GSM8K | accuracy: 80.53 -> 81.06 | Helped |
| increase generation length from 256 to 384 without further RL fine-tuning (ablation) | LLaDOU · MATH | accuracy: 44.6 -> 46.2 | Helped |
| increase generation length from 384 to 448 without further RL fine-tuning (ablation) | LLaDOU · MATH | accuracy: 46.2 -> 45.7 | Hurt |
| increase generation length from 448 to 512 without further RL fine-tuning (ablation) | LLaDOU · MATH | accuracy: 45.7 -> 45.9 | Helped |
Surprising findings
- Training only the unmasking policy module with LLaDA frozen gives a large gain (47.27% to 69.24% on GSM8K), suggesting the unmasking order is itself a key lever.
- DCoLT lifts Dream-7B on GSM8K from 50.11% to 80.53% in the appendix, comparable to the LLaDA gain.
Limitations
- Performance on math and code remains below larger proprietary-data models due to limited training data and compute.
- Only validated on tasks with verifiable rewards; a reward model would be needed for general tasks.
- Long-generation gains depend on fine-tuning with longer lengths; results at other lengths are evaluated without RL on those lengths.
Experimental setup
Datasets: GSM8K MATH KodCode-V1-SFT-R1 GSM8K-Aug Sudoku 4x4 (generated)
Benchmarks: GSM8K MATH MBPP HumanEval Sudoku 4x4 GSM8K-Aug
Metrics: accuracy pass@1
Baselines: GPT2 + CoT GPT2 + Implicit CoT SEDD + DoT LLaDA 8B diffu-GRPO d1-LLaDA Dream 7B DeepSeekMath-RL 7B DeepSeekCoder-Instruct 7B
Builds on: GRPO SEDD LLaDA Chain-of-Thought diffu-GRPO
Closest papers by meaning
- Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models (ICLR 2026 · similarity 0.90)
- Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models (ICML 2026 · similarity 0.89)
- d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via Monotonic Entropy Descent with Reinforcement Learning (ICML 2026 · similarity 0.88)
- Diffusion of Thought: Chain-of-Thought Reasoning in Diffusion Language Models (NeurIPS 2024 · similarity 0.88)
- d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.