Research map / Reinforcement learning / RL from feedback

Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes R2PO, a two-stage LLM policy search where a Search-LLM proposes linear/tabular policy parameters and a Critic-LLM revises them using median-trajectory evidence, aggregate rollout statistics, and a revision rule, with keep-best selection; identifies "salience bias" in multi-trajectory critiques.

Problem

LLM-based policy optimizers see only scalar rewards and cannot diagnose why a policy fails; naively giving trajectories to an LLM reviser can cause high-variance regressions.

Approach

Separate global LLM search over policy parameters from a trajectory-grounded Critic-LLM revision, feeding the critic a median rollout plus aggregate statistics and an abstention rule, and keep the better of original and revised candidates.

Key results

• R2PO achieves highest mean reward on 9 of 10 environments (e.g., CartPole 474.67 ± 16.90, InvertedPendulum 756.08 ± 154.50), tied-second on MountainCarContinuous where SAC leads (82.33 vs 81.61).
• In a three-trajectory variant (ThreeTraj) on CartPole, salience-problem regressions account for 76.6% of regressions (233 of 304).
• AlwaysCritic regressions were 34% on CartPole and 31.6% on Pong; R2PO has 13.7% regression rate on CartPole vs 30.4% for ThreeTraj.

Surprising findings

• Giving the Critic-LLM more trajectories (ThreeTraj) performs worse than a single representative trajectory, due to salience bias on the worst rollout.
• CriticOnly with a rich prompt but no real rollouts is the strongest single-factor ablation, yet still underperforms R2PO.

Limitations

• Restricted to compact policy classes (linear and tabular); high-dimensional neural policies untested.
• Requires an environment-dependent revision threshold tau_c with coarse task-specific calibration.
• Salience bias analysis relies on coded reasoning text and a strict definition of regressions.

Experimental setup

Datasets: CartPole-v1 FrozenLake-v1 MountainCar-v0 MountainCarContinuous-v0 InvertedPendulum-v5 InvertedDoublePendulum-v5 Swimmer-v5 Maze Nim Pong

Benchmarks: CartPole-v1 FrozenLake-v1 MountainCar-v0 MountainCarContinuous-v0 InvertedPendulum-v5 InvertedDoublePendulum-v5 Swimmer-v5 Maze Nim Pong

Metrics: mean reward mean best reward stability gap Welch t-test p-value

Baselines: ProPS ProPS+ PPO TRPO SAC DQN A2C DDPG TD3

Builds on: ProPS OPRO Reflexion Self-Refine

Closest papers by meaning

• Reflective Policy Optimization (ICML 2024 · similarity 0.89)
• Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs (NeurIPS 2025 · similarity 0.89)
• ContraDiff: Planning Towards High Return States via Contrastive Learning (ICLR 2025 · similarity 0.88)
• Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.88)
• PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization (NeurIPS 2026 · similarity 0.88)
• Calibration Is Not Control: Intervention Advantage for LLM-Agent Oversight (NeurIPS 2026 · similarity 0.88)