Research map / Reinforcement learning / Policy gradient and actor-critic
Off-Policy Proximal Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use replayed off-policy data with the clipped surrogate objective instead of using only on-policy data | PPO · Walker2d, Ant, and Humanoid v2 | sample efficiency | Helped |
| Use replayed off-policy data with the clipped surrogate objective instead of using only on-policy data | PPO · most continuous control tasks | final return | Helped |
| Use the scaled clipping bounds in the off-policy clipped surrogate objective | Off-Policy PPO · continuous control tasks | policy-update KL divergence | No clear effect |
| Use Off-Policy PPO instead of SOP | continuous control agents · Hopper v2 | timesteps to reach threshold: 355 -> 210 | Helped |
| Use Off-Policy PPO instead of TD3 | continuous control agents · Walker2d v2 | timesteps to reach threshold: 285 -> 220 | Helped |
| Use Off-Policy PPO instead of TD3 | continuous control agents · HalfCheetah v2 | timesteps to reach threshold: 45 -> 70 | Hurt |
| Use Off-Policy PPO instead of SOP | continuous control agents · Ant v2 | timesteps to reach threshold: 285 -> 310 | Hurt |
| Use Off-Policy PPO instead of TD3 | continuous control agents · Humanoid v2 | timesteps to reach threshold: 950 -> 2800 | Hurt |
Closest papers by meaning
- The Sufficiency of Off-Policyness and Soft Clipping: PPO Is Still Insufficient according to an Off-Policy Measure (AAAI 2023 · similarity 0.93)
- PPO in the Fisher-Rao geometry (NeurIPS 2026 · similarity 0.90)
- Simple Policy Optimization (ICML 2025 · similarity 0.89)
- Colored Noise in PPO: Improved Exploration and Performance through Correlated Action Sampling (AAAI 2024 · similarity 0.87)
- Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol (NeurIPS 2025 · similarity 0.87)
- Adversarial Policy Optimization for Offline Preference-based Reinforcement Learning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.