Research map / Reinforcement learning / Policy gradient and actor-critic
Absolute Policy Optimization: Enhancing Lower Probability Bound of Performance with High Confidence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Absolute Policy Optimization (APO), a trust-region on-policy method that maximizes a lower probability bound of return (J - kV) with high confidence, plus a PPO-style variant PAPO, evaluated on GUARD continuous control, MuJoCo/Gymnasium tasks, and 62 Atari games.
Problem
Standard trust-region on-policy RL (TRPO/PPO) maximizes expected return and gives no control over worst-case performance samples; the paper targets improving a high-confidence lower probability bound of performance.
Approach
Defines the lower bound B_k = J(pi) - k V(pi) (variance-based Selberg/Chebyshev-type inequality) and derives a surrogate with monotonic improvement guarantees so that a trust-region method optimizes this bound; TRPO is the special case k=0.
Key results
- APO obtained the highest expected and worst-case reward counts across Atari and GUARD in the stated count table (e.g., 26 of 62 Atari games best on average expected reward over all epochs vs 22 for PPO).
- APO adds under 5% wall-clock time over TRPO on Goal_Ant according to the authors' computational cost comparison.
- PAPO outperforms PPO and other proximal baselines on high-dimensional GUARD and HumanoidStandup/HandReach tasks per the authors' curves.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use APO instead of PPO | Atari · Atari, average expected reward over all epochs | number of highest evaluation scores: 26 vs 22 | Helped |
| Use APO instead of PPO | Atari · Atari, average expected reward over last 100 epochs | number of highest evaluation scores: 29 vs 17 | Helped |
| Use APO instead of PPO | GUARD · GUARD, average expected reward over all epochs | number of highest evaluation scores: 9 vs 3 | Helped |
| Use APO instead of PPO | GUARD · GUARD, average expected reward over last 20 epochs | number of highest evaluation scores: 9 vs 2 | Helped |
| Use APO instead of PPO | Atari · Atari, average worst reward over all epochs | number of highest evaluation scores: 26 vs 24 | Helped |
| Use APO instead of PPO | Atari · Atari, average worst reward over last 100 epochs | number of highest evaluation scores: 27 vs 20 | Helped |
| Use APO instead of PPO | GUARD · GUARD, average worst reward over all epochs | number of highest evaluation scores: 8 vs 2 | Helped |
| Use APO instead of TRPO | GUARD · GUARD, average worst reward over last 20 epochs | number of highest evaluation scores: 9 vs 2 | Helped |
| Treat |H(s,a,s')|_max as a hyperparameter rather than computing it from the recent policy (ablation) | APO · Riverraid and PUSH_Ant | reward | Helped |
| Use a moderate probability factor k rather than a very small or excessively large k (ablation) | APO · Riverraid | overall improvement | Mixed |
| Add clipped-surrogate early stopping to APO (PAPO) | APO · Challenging continuous GUARD tasks | converged reward | Helped |
Surprising findings
- Treating the bound's ||mu^T||_inf and |H|_max terms as tuned hyperparameters rather than computed quantities improved performance, departing from the theory.
Limitations
- On-policy sample inefficiency inherited from trust-region methods.
- Practical implementation departs from theory (hyperparameter substitutes, averaging over states instead of max), weakening the guarantee.
- Evaluated only on small MLP policies (two 64-unit tanh layers) and fixed-environment settings.
Experimental setup
Datasets: GUARD MuJoCo Gymnasium Robotics Atari (62 games, ALE)
Benchmarks: GUARD Atari 62 games MuJoCo Humanoid HandReach HandManipulateEgg HandManipulateBlock
Metrics: average expected reward over all epochs average expected reward over last epochs average worst reward normalized score vs TRPO wall-clock time GPU/CPU usage
Baselines: TRPO PPO A2C AlphaPPO ESPO V-MPO
Builds on: TRPO PPO Selberg's inequality GUARD benchmark
Closest papers by meaning
- Simple Policy Optimization (ICML 2025 · similarity 0.88)
- Augmented Proximal Policy Optimization for Safe Reinforcement Learning (AAAI 2023 · similarity 0.88)
- Learning to Constrain Policy Optimization with Virtual Trust Region (NeurIPS 2022 · similarity 0.87)
- Adversarial Policy Optimization for Offline Preference-based Reinforcement Learning (ICLR 2025 · similarity 0.86)
- ACPO: A Policy Optimization Algorithm for Average MDPs with Constraints (ICML 2024 · similarity 0.86)
- Reflective Policy Optimization (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.