Research map / Reinforcement learning / Policy gradient and actor-critic
The Sufficiency of Off-Policyness and Soft Clipping: PPO Is Still Insufficient according to an Off-Policy Measure
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace PPO clipping with the sigmoidally preconditioned Scopic objective and KL regularization | PPO, yielding P3O · Six Gym and Arcade Learning Environment tasks | episodic accumulated reward | Helped |
| Use the Scopic objective instead of PPO clipping | P3O and PPO · All tasks except BeamRider | CPI objective | Helped |
| Use the Scopic objective instead of PPO clipping | P3O and PPO · BeamRider | CPI objective | Mixed |
| Use sigmoid preconditioning without clipping | P3O · Training across the evaluated tasks | DEON: up to as big as 60.0 | Helped |
| Use a decaying learning rate | P3O · Discrete tasks | DEON | No clear effect |
| Use a fixed learning rate rather than a decaying schedule | P3O · Continuous tasks | DEON | Helped |
| Use a fixed learning rate rather than a decaying schedule (ablation) | P3O · HalfCheetah | episodic accumulated reward | Helped |
| Use a decaying learning rate rather than a fixed schedule (ablation) | PPO · HalfCheetah | episodic accumulated reward | Helped |
| Use the best tested hyperparameter group (ablation) | P3O · HalfCheetah | episodic accumulated reward | Helped |
| Use the best tested hyperparameter group (ablation) | PPO · HalfCheetah | episodic accumulated reward | Helped |
| Use the best tested P3O hyperparameter group rather than the best PPO group (ablation) | HalfCheetah · HalfCheetah | episodic accumulated reward | Helped |
Closest papers by meaning
- Off-Policy Proximal Policy Optimization (AAAI 2023 · similarity 0.93)
- On Stationary Point Convergence of PPO-Clip (ICLR 2024 · similarity 0.90)
- PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clipping (AAAI 2024 · similarity 0.89)
- PPO in the Fisher-Rao geometry (NeurIPS 2026 · similarity 0.88)
- Towards Convergence of PPO: An Approximate Descent Approach (NeurIPS 2026 · similarity 0.88)
- Simple Policy Optimization (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.