Research map / Reinforcement learning / RL from feedback
Benefits and Pitfalls of Reinforcement Learning for Language Model Planning: A Theoretical Perspective
Siwei Wang, Yifei Shen, Haoran Sun, Shi Feng, Shang-Hua Teng, Li Dong, Yaru Hao, Wei Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| continue SFT on the same data instead of switching to policy gradient (ablation) | SFT base model · Synthetic path planning (ER graph, 100 nodes) | test accuracy | Hurt |
| use policy gradient with on-policy exploration instead of SFT | SFT base model · Synthetic path planning (ER graph, 100 nodes) | test accuracy | Improved |
| train PG without KL regularization (ablation) | Policy gradient · Synthetic path planning | output diversity: diversity falls until one path per pair | Hurt |
| continue PG training after diversity collapses (ablation) | Policy gradient without KL · Synthetic path planning | test accuracy | Hurt |
| add KL regularization to PG (higher lambda) (ablation) | Policy gradient · Synthetic path planning | output diversity | Improved |
| add KL regularization to PG (higher lambda) (ablation) | Policy gradient · Synthetic path planning | train accuracy | Hurt |
| use only outcome reward in Q-learning (ablation) | Q-learning · Synthetic path planning | train and test accuracy: collapses to near-zero accuracy | Hurt |
| add process rewards (target check and adjacency check) to Q-learning (ablation) | Q-learning · Synthetic path planning | test accuracy | Improved |
| use off-policy sampling with process-reward Q-learning (ablation) | Q-learning · Synthetic path planning | train and test accuracy | No clear effect |
| replace PG with Q-learning using process rewards | Policy gradient · Synthetic path planning | output diversity at high accuracy | Improved |
Closest papers by meaning
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.92)
- Representation-Based Exploration for Language Models: From Test-Time to Post-Training (ICLR 2026 · similarity 0.91)
- Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.90)
- ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.90)
- On the Modeling Capabilities of Large Language Models for Sequential Decision Making (ICLR 2025 · similarity 0.90)
- Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
