Research map / Reinforcement learning / Policy gradient and actor-critic
Improving Value Estimation Critically Enhances Vanilla Policy Gradient
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that vanilla policy gradient matches or beats PPO on MuJoCo continuous control simply by increasing the number of value update steps per iteration, arguing value estimation accuracy, not trust regions, drives on-policy performance.
Problem
Why do modern policy gradient methods (TRPO, PPO) outperform vanilla policy gradient (VPG) in continuous control, given that trust-region theory does not hold in practice and objective landscapes are non-smooth/fractal?
Approach
Attributes PPO's and TRPO's advantage over VPG mainly to more value-network gradient steps per iteration, which yield more accurate value/advantage estimates; simply increasing VPG's value steps (e.g., to 50-100) closes the gap with PPO, and a Lyapunov/Hölder-continuity argument suggests value networks need more updates than policy networks.
Key results
- VPG with enough value steps (about 50 or more) matches or outperforms PPO on Gymnasium MuJoCo tasks, including Ant and Humanoid.
- Default PPO with 10 epochs consistently violates its ratio clipping bound, while VPG and 1-epoch PPO stay within it yet perform worse.
- With GAE lambda=1, PPO's reward drops by more than 40% in Walker and HalfCheetah, while VPG with 100 value steps is only slightly affected.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase VPG value updates per iteration (up to 50 or more) (ablation) | VPG · HalfCheetah, Hopper, Walker, Ant, and Humanoid | cumulative reward: 50 | Helped |
| Set GAE λ from 0.95 to 1 (ablation) | VPG · Hopper | cumulative reward: 2601.57 ± 232.14 -> 2323.28 ± 436.24 | Hurt |
| Set GAE λ from 0.95 to 1 (ablation) | VPG · Walker | cumulative reward: 3457.79 ± 646.70 -> 3123.83 ± 987.58 | Hurt |
| Set GAE λ from 0.95 to 1 (ablation) | VPG · HalfCheetah | cumulative reward: 4928.88 ± 807.04 -> 4381.30 ± 222.15 | Hurt |
| Set GAE λ from 0.95 to 1 (ablation) | PPO · Hopper | cumulative reward: 1965.29 ± 478.14 -> 1611.46 ± 541.32 | Hurt |
| Set GAE λ from 0.95 to 1 (ablation) | PPO · Walker | cumulative reward: 2527.74 ± 507.40 -> 1431.50 ± 612.17 | Hurt |
| Set GAE λ from 0.95 to 1 (ablation) | PPO · HalfCheetah | cumulative reward: 4488.60 ± 2699.54 -> 2604.08 ± 1237.80 | Hurt |
| Add reward normalization (ablation) | VPG with 1 or 10 value steps · Continuous-control benchmarks | cumulative reward | Helped |
| Add reward normalization (ablation) | VPG with 100 value steps · Continuous-control benchmarks | cumulative reward | No clear effect |
| Use full-batch updates instead of mini-batches (ablation) | PPO · Hopper | cumulative reward | Hurt |
| Use mini-batch size 512 and 2 epochs (ablation) | PPO · Ant and Humanoid | cumulative reward: 512, # epoch = 2 | Helped |
| Change policy learning rate (ablation) | PPO · Ant | cumulative reward | Mixed |
Surprising findings
- PPO's ratio clipping does not keep probability ratios inside the trust region in practice, and enforcing the trust region does not correlate with policy improvement.
- Full-batch PPO with the same number of epochs fails to find a good policy, attributed to under-optimized value estimation.
Limitations
- Analysis is confined to continuous control; the authors refrain from claiming validity for LLMs or other domains.
- The theoretical result relies on Lipschitz/Hölder assumptions and a Lyapunov-exponent argument that is only loosely connected to the empirical experiments.
Experimental setup
Datasets: Gymnasium MuJoCo environments (Hopper, Walker, HalfCheetah, Ant, Humanoid)
Benchmarks: Hopper Walker2d HalfCheetah Ant Humanoid
Metrics: cumulative reward value estimation error
Baselines: PPO TRPO VPG with 1 value step PPO with 1 epoch
Builds on: PPO TRPO REINFORCE Generalized Advantage Estimation CleanRL
Closest papers by meaning
- A Parametric Class of Approximate Gradient Updates for Policy Optimization (ICML 2022 · similarity 0.90)
- A Closer Look at Deep Policy Gradients (ICLR 2020 · similarity 0.89)
- Stabilizing Policy Gradient Methods via Reward Profiling (AAAI 2026 · similarity 0.87)
- Implementation Matters in Deep RL: A Case Study on PPO and TRPO (ICLR 2020 · similarity 0.87)
- Improving Deep Policy Gradients with Value Function Search (ICLR 2023 · similarity 0.87)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.