Research map / Reinforcement learning / Policy gradient and actor-critic
A Closer Look at Deep Policy Gradients
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the learned value network instead of a zero baseline | PPO/TRPO gradient estimation · Walker2d-v2 and Hopper-v2 | gradient variance | Helped |
| Use the true value function instead of the learned value network | PPO/TRPO gradient estimation · Walker2d-v2 and Hopper-v2 | gradient variance | Helped |
| Remove the baseline (use returns rather than advantages) | PPO/TRPO agents · MuJoCo tasks | agent reward: almost an order of magnitude worse reward | Hurt |
| Estimate gradients with the standard practical sample regime | PPO/TRPO · MuJoCo tasks | cosine similarity to the true gradient | Hurt |
| Increase task complexity | PPO/TRPO gradient estimation · MuJoCo tasks | gradient estimate quality | Hurt |
| Continue training | PPO/TRPO gradient estimation · MuJoCo tasks | gradient estimate quality | Hurt |
| Increase the number of samples used to estimate the true reward landscape | PPO/TRPO · Humanoid-v2 | reward landscape smoothness | Helped |
| Increase training progress | PPO surrogate objective · Humanoid-v2 | alignment with true reward | Hurt |
| Use the TRPO surrogate-objective update direction in the higher-sample regime | TRPO · Humanoid-v2 | alignment with true reward | Helped |
| Use the PPO surrogate-objective update direction, even in the higher-sample regime | PPO · Humanoid-v2 | true reward | Hurt |
| Train the value network on its GAE-based supervised loss | PPO/TRPO value networks · MuJoCo tasks | MRE on the training loss target | Helped |
| Use the GAE-based value-training loss as a proxy for learning the true value function | PPO/TRPO value networks · MuJoCo tasks | MRE to the true value function: about 50% | Hurt |
Closest papers by meaning
- Improving Deep Policy Gradients with Value Function Search (ICLR 2023 · similarity 0.90)
- A Parametric Class of Approximate Gradient Updates for Policy Optimization (ICML 2022 · similarity 0.90)
- Improving Value Estimation Critically Enhances Vanilla Policy Gradient (ICML 2025 · similarity 0.89)
- Mollification Effects of Policy Gradient Methods (ICML 2024 · similarity 0.88)
- Risk-Sensitive Policy Optimization via Predictive CVaR Policy Gradient (ICML 2024 · similarity 0.87)
- Identifying Policy Gradient Subspaces (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.