Research map / Reinforcement learning / Policy gradient and actor-critic
Reflective Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Reflective Policy Optimization (RPO), an on-policy PPO extension whose clipped surrogate also uses the ratio and advantage of the subsequent state-action pair, giving a tighter lower bound and better MuJoCo and Atari sample efficiency.
Problem
On-policy RL such as PPO is sample inefficient because policy updates use only the current state-action pair and ignore the impact of subsequent states and actions in the trajectory.
Approach
Extends the TRPO/PPO surrogate with a k=2 generalized term that multiplies two consecutive importance ratios with the subsequent advantage, removes the problematic "1" term from the generalized surrogate, and clips each ratio separately, claiming monotonic improvement and a shrinking solution space.
Key results
- On MuJoCo, RPO reaches HalfCheetah 3495, Hopper 3262, Walker2d 4025, Humanoid 5793 mean return, above PPO (2408, 2795, 3643, 5608).
- On 54 Atari games, RPO vs PPO normalized improvement averages 74.3% (median 15.3%), improving 41 of 54 games.
- In CliffWalking, RPO reduces cliff falls to 251 versus 328 for PPO (k=3 variant: 257).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RPO instead of GePPO | on-policy RL · HalfCheetah | average return: 3153 -> 3495 | Helped |
| use RPO instead of GePPO | on-policy RL · Hopper | average return: 3170 -> 3262 | Helped |
| use RPO instead of PPO | on-policy RL · Reacher | average return: -5.2 -> -4.9 | Helped |
| use RPO instead of GePPO | on-policy RL · Walker2d | average return: 3690 -> 4025 | Helped |
| use RPO instead of GePPO | on-policy RL · Swimmer | average return: 157 -> 157 | No clear effect |
| use RPO instead of PPO | on-policy RL · Humanoid | average return: 5608 -> 5793 | Helped |
| use RPO instead of PPO | RPO · CliffWalking training | cliff falls: 328 -> 257 | Helped |
| add a third trajectory ratio (k = 3) (ablation) | RPO · CliffWalking training | cliff falls: 257 -> 251 | Helped |
| add a third trajectory ratio (k = 3) (ablation) | RPO · MuJoCo environments | average return | Mixed |
| clip the two ratios together instead of clipping each separately (ablation) | RPO · HalfCheetah and Swimmer | average return | Hurt |
| reduce the clipping values (ablation) | RPO · MuJoCo environments | average return | Helped |
| reduce the weighting coefficient alongside certain clipping levels (ablation) | RPO · MuJoCo environments | average return | Helped |
Surprising findings
- Removing the constant '1' from the generalized surrogate is reported to make TayPO perform very poorly (e.g., HalfCheetah -137), which the authors attribute to the '1' term.
Limitations
- Higher k increases variance of the product of ratios, so only k=2 is used in main experiments.
- Hyperparameters (beta, epsilon1) are tuned per domain and not fully explored across environments.
- Evaluated only on MuJoCo, Atari, and a toy CliffWalking; no LLM or large-scale tasks.
Experimental setup
Datasets: MuJoCo Atari 2600 CliffWalking
Benchmarks: MuJoCo Atari 2600 CliffWalking
Metrics: average return normalized improvement over PPO cliff fall count episode length
Baselines: TRPO PPO ISPO TayPO GePPO OTRPO
Builds on: PPO TRPO TayPO GePPO
Closest papers by meaning
- Mirror Descent Policy Optimization (ICLR 2022 · similarity 0.91)
- Simple Policy Optimization (ICML 2025 · similarity 0.89)
- Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias (NeurIPS 2026 · similarity 0.89)
- Embedding Safety into RL: A New Take on Trust Region Methods (ICML 2025 · similarity 0.87)
- ACPO: A Policy Optimization Algorithm for Average MDPs with Constraints (ICML 2024 · similarity 0.87)
- Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.