Research map / Reinforcement learning / Policy gradient and actor-critic
Improving Deep Policy Gradients with Value Function Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply two-scale VFS critic search | PPO, DDPG, and TD3 · five MuJoCo continuous-control tasks | average return: 26% return improvement (PPO); 29% return improvement (DDPG); 19% return improvement (TD3) | Helped |
| Apply VFS critic search | PPO, DDPG, and TD3 · five MuJoCo continuous-control tasks | steps to reach baseline peak performance: 33% fewer samples (PPO); 35% improvement in sample efficiency (DDPG); 43% fewer samples (TD3) | Helped |
| Use two-scale perturbations rather than small-scale perturbations alone (ablation) | VFS-PPO · Hopper | return | Helped |
| Use two-scale perturbations rather than big-scale perturbations alone (ablation) | VFS-PPO · Hopper | return | Helped |
| Remove big-scale perturbations (ablation) | VFS-PPO · Hopper | return | Hurt |
| Remove small-scale perturbations (ablation) | VFS-PPO · Hopper | gradient estimate variance | Hurt |
| Anneal perturbations linearly to zero toward step $9 \times 10^5$ (ablation) | VFS-PPO · Hopper | return | Hurt |
| Increase population size beyond 10 (ablation) | VFS-PPO · Hopper | return | No clear effect |
| Increase population size (ablation) | VFS-PPO · Hopper | return | Helped |
| Use extra critic gradient steps instead of population search (ablation) | PPO · Hopper | return | No clear effect |
| Use small-scale perturbations (ablation) | VFS-PPO · Hopper | gradient estimate similarity | Mixed |
| Use big-scale perturbations (ablation) | VFS-PPO · Hopper | gradient estimate similarity | Mixed |
Closest papers by meaning
- Learning Value Functions in Deep Policy Gradients using Residual Variance (ICLR 2021 · similarity 0.91)
- A Closer Look at Deep Policy Gradients (ICLR 2020 · similarity 0.90)
- Improving Value Estimation Critically Enhances Vanilla Policy Gradient (ICML 2025 · similarity 0.87)
- Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (ICML 2024 · similarity 0.87)
- Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn (NeurIPS 2024 · similarity 0.87)
- Value Improved Actor Critic Algorithms (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.