Research map / Reinforcement learning / Policy gradient and actor-critic
Ordered Policy Optimization
Dongkyu (Derek) Cho, Pan Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Ordered Policy Optimization, a PPO variant that adapts the clipping radius per sample instead of using one fixed clipping range for all importance ratios.
Problem
PPO's fixed clipping range on the importance ratio treats all samples alike, so it cannot distinguish statistically harmful extreme ratios from large but tolerable policy changes.
Approach
Adapt the clipping radius to the statistical behavior of each sample's importance ratio, rather than using one fixed range, so extreme harmful ratios are clipped while large tolerable changes are retained.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace fixed PPO clipping with order-statistic sample-wise radii (OPO) | PPO · MuJoCo (6 tasks, 5 seeds) | normalized aggregate score (median, IQM, mean, optimality gap) | Improved |
| Replace fixed PPO clipping with OPO | PPO · Atari (ResNet-18, 3 seeds) | relative improvement over PPO | Improved |
| Use OPO instead of PPO and SPO on RoadRunner, NameThisGame, BeamRider | PPO / SPO · Atari learning curves | final return | Improved |
| Set prescribed GPD teacher shape k*=0.33 instead of larger k (0.49, 0.70) (ablation) | OPO · MuJoCo | performance | Improved |
| Increase prescribed teacher shape k (ablation) | OPO · MuJoCo ratio tail | tail decay of policy ratio | Hurt |
| Use pointwise clipping to bound ratios | PPO · MuJoCo | ratio tail heaviness | No clear effect |
Experimental setup
Baselines: PPO
Builds on: PPO
Closest papers by meaning
- Towards Convergence of PPO: An Approximate Descent Approach (NeurIPS 2026 · similarity 0.89)
- Off-Policy Proximal Policy Optimization (AAAI 2023 · similarity 0.88)
- Simple Policy Optimization (ICML 2025 · similarity 0.85)
- PPO in the Fisher-Rao geometry (NeurIPS 2026 · similarity 0.85)
- PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clipping (AAAI 2024 · similarity 0.85)
- Preference Optimization by Estimating the Ratio of the Data Distribution (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
