Research map / Reinforcement learning / RL from feedback
On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning
Yifan Zhang, Yifeng Liu, Huizhuo Yuan, Yang Yuan, Quanquan Gu, Andrew C Yao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RPG-UFKL instead of DAPO | DAPO · AIME24 (8K, last) | Mean@32 accuracy: 0.5438 -> 0.5938 | Improved |
| use RPG-REINFORCE-URKL instead of DAPO | DAPO · AIME25 (8K, last) | Mean@32 accuracy: 0.4469 -> 0.5073 | Improved |
| use RPG-REINFORCE-UFKL instead of DAPO | DAPO · AIME25 (8K, best) | Mean@32 accuracy: 0.4740 -> 0.5031 | Improved |
| use RPG-UFKL instead of GRPO | GRPO · AMC23 (8K, last) | Mean@32 accuracy: 0.9109 -> 0.9492 | Improved |
| use RPG-UFKL instead of GRPO | GRPO · AIME24 (8K, last) | Mean@32 accuracy: 0.3750 -> 0.5938 | Improved |
| use reverse KL (URKL) instead of forward KL (UFKL) in fully differentiable RPG (ablation) | RPG · AIME24 (8K, last) | Mean@32 accuracy: 0.5938 -> 0.4542 | Hurt |
| use reverse KL (URKL) instead of forward KL (UFKL) in fully differentiable RPG (ablation) | RPG · AIME25 (8K, last) | Mean@32 accuracy: 0.4698 -> 0.5261 | Improved |
| use REINFORCE-style loss instead of fully differentiable loss (URKL) (ablation) | RPG-URKL · AIME24 (8K, last) | Mean@32 accuracy: 0.4542 -> 0.5708 | Improved |
| use REINFORCE-style loss instead of fully differentiable loss (UFKL) (ablation) | RPG-UFKL · AIME24 (8K, last) | Mean@32 accuracy: 0.5938 -> 0.5906 | No clear effect |
| change clip ratios from (0.1,0.1) to (0.2,0.28) (ablation) | REINFORCE-style RPG · Qwen3-4B training | Mean@32 accuracy | No clear effect |
| use DAPO clip-higher (ablation) | RPG-REINFORCE · Qwen3-4B training | Mean@32 accuracy | Mixed |
Closest papers by meaning
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.91)
- KL for a KL: On-Policy Distillation with Control Variate Baseline (NeurIPS 2026 · similarity 0.90)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.90)
- Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.90)
- SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning (NeurIPS 2026 · similarity 0.89)
- Hybrid Policy Distillation for LLMs (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
