Research map / Reinforcement learning / RL from feedback
Simple Policy Gradients for Reasoning with Diffusion Language Models
Anthony Zhan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use stepwise denoising-MDP policy gradient (AGRPO) instead of ELBO-based sequence-level objective (SPG) | LLaDA-8B-Instruct · Countdown (len 512) | accuracy (%): 70.3 -> 82.4 | Improved |
| use AGRPO instead of SPG | LLaDA-8B-Instruct · GSM8K (len 512) | accuracy (%): 84.5 -> 88.1 | Improved |
| use AGRPO instead of SPG | LLaDA-8B-Instruct · MATH-500 (len 256) | accuracy (%): 40.0 -> 39.4 | Hurt |
| use AGRPO instead of SPG | LLaDA-8B-Instruct · MATH-500 (len 512) | accuracy (%): 41.8 -> 42.0 | No clear effect |
| use AGRPO instead of SPG | LLaDA-8B-Instruct · Sudoku (len 512) | accuracy (%): 93.1 -> 95.2 | Improved |
| post-train with AGRPO then sample with few steps (m=32) | LLaDA-8B-Instruct · GSM8K test, n=384 | accuracy (%): 59.7% at 12 tokens per step | Improved |
| post-train with AGRPO to reduce sampling steps | LLaDA-8B-Instruct · GSM8K test, n=384 | sampling steps for equal accuracy: 4x fewer steps | Improved |
| increase MC timestep samples k from 2 to 32 (ablation) | AGRPO · GSM8K training | convergence speed / wall time: 44.8s -> 72.4s per step | Mixed |
| increase k from 32 to 128 (ablation) | AGRPO · GSM8K training | convergence speed / wall time: 72.4s -> 158.1s per step | No clear effect |
| add low-discrepancy and entropy importance sampling (ablation) | AGRPO · Countdown, n=128, m=64 | reward convergence and gradient norm | Improved |
| add KL penalty (beta>0) (ablation) | AGRPO · reasoning tasks | accuracy | No clear effect |
| use LoRA fine-tuned AGRPO vs GRPO on AR model | AGRPO · GSM8K (len 512) | accuracy gain over base: +9.9% vs +5.3% for GRPO | Improved |
Closest papers by meaning
- d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
- Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages (NeurIPS 2026 · similarity 0.91)
- Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via Monotonic Entropy Descent with Reinforcement Learning (ICML 2026 · similarity 0.90)
- d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation (ICML 2026 · similarity 0.90)
- The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models (ICML 2026 · similarity 0.90)
- Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
