Research map / Reinforcement learning / RL from feedback
dTRPO : Trajectory Reduction in Policy Optimization of Diffusion Large Language Models
Wenxuan Zhang, Lemeng Wu, Changsheng Zhao, Ernie Chang, Mingchen Zhuge, Zechun Liu, DiJia Su, Hanxian Huang, Jun Chen, Chong Zhou, Raghuraman Krishnamoorthi, Vikas Chandra and 2 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use dTRPO trajectory-reduction DPO instead of the base model | Fast-dLLM-v2-7B · GPQA | accuracy: 20.71 -> 30.30 | Improved |
| use dTRPO instead of the base model | Fast-dLLM-v2-7B · MATH | accuracy: 60.26 -> 64.3 | Improved |
| use dTRPO instead of the base model | Fast-dLLM-v2-7B · Arena-Hard v2.0 | score: 1.26 -> 7.41 | Improved |
| use dTRPO instead of VRPO (ELBO-based DPO) | Fast-dLLM-v2-7B · LiveCodeBench v6 | accuracy: 12.89 -> 15.17 | Improved |
| use dTRPO instead of DPO with mean-field estimate | Fast-dLLM-v2-7B · IFEval | accuracy: 65.62 -> 65.06 | No clear effect |
| use ratios of newly unmasked tokens (dTRPO) instead of directly maximizing masked-token probability (ablation) | Fast-dLLM-v2-7B · GSM8K | accuracy: 66.26 -> 85.97 | Improved |
| use dTRPO instead of ELBO objective on masked tokens (ablation) | Fast-dLLM-v2-7B · IFEval | accuracy: 51.02 -> 65.06 | Improved |
| mix math and code preference data with SmolTalk2 (ablation) | dTRPO · LiveCodeBench v6 | accuracy: 13.36 -> 15.17 | Improved |
| mix math and code preference data with SmolTalk2 (ablation) | dTRPO · IFEval | accuracy: 67.10 -> 65.06 | Hurt |
| use on-policy data instead of mixed off-policy data (ablation) | dTRPO · MATH and code | accuracy: 64.30 -> 62.74 (MATH) | Hurt |
| apply dTRPO to long-block LLaDA | LLaDA · MATH (256 tokens) | accuracy: 27.84 -> 34.88 | Improved |
| apply dTRPO to Fast-dLLM-v2 1.5B | Fast-dLLM-v2 1.5B · MATH | accuracy: 37.9 -> 38.96 | No clear effect |
Closest papers by meaning
- d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation (ICML 2026 · similarity 0.91)
- d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models (ICLR 2026 · similarity 0.89)
- Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages (NeurIPS 2026 · similarity 0.89)
- Diffusion-State Policy Optimization for Masked Diffusion Language Models (NeurIPS 2026 · similarity 0.89)
- Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via Monotonic Entropy Descent with Reinforcement Learning (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
