AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

Benefits and Pitfalls of Reinforcement Learning for Language Model Planning: A Theoretical Perspective

Siwei Wang, Yifei Shen, Haoran Sun, Shi Feng, Shang-Hua Teng, Li Dong, Yaru Hao, Wei Chen

ICLR 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
continue SFT on the same data instead of switching to policy gradient (ablation)SFT base model · Synthetic path planning (ER graph, 100 nodes)test accuracyHurt
use policy gradient with on-policy exploration instead of SFTSFT base model · Synthetic path planning (ER graph, 100 nodes)test accuracyImproved
train PG without KL regularization (ablation)Policy gradient · Synthetic path planningoutput diversity: diversity falls until one path per pairHurt
continue PG training after diversity collapses (ablation)Policy gradient without KL · Synthetic path planningtest accuracyHurt
add KL regularization to PG (higher lambda) (ablation)Policy gradient · Synthetic path planningoutput diversityImproved
add KL regularization to PG (higher lambda) (ablation)Policy gradient · Synthetic path planningtrain accuracyHurt
use only outcome reward in Q-learning (ablation)Q-learning · Synthetic path planningtrain and test accuracy: collapses to near-zero accuracyHurt
add process rewards (target check and adjacency check) to Q-learning (ablation)Q-learning · Synthetic path planningtest accuracyImproved
use off-policy sampling with process-reward Q-learning (ablation)Q-learning · Synthetic path planningtrain and test accuracyNo clear effect
replace PG with Q-learning using process rewardsPolicy gradient · Synthetic path planningoutput diversity at high accuracyImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy