Research map / Reinforcement learning / Policy gradient and actor-critic
Value Improved Actor Critic Algorithms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Value-Improved Actor Critic (VIAC), decoupling the acting policy from the critic's evaluated policy so the evaluated policy gets greedier updates (e.g., expectile-based implicit improvement), with convergence theory in finite-horizon MDPs and gains on TD3/SAC in DeepMind control.
Problem
Actor-critic methods that improve a DNN-parameterized acting policy with gradient steps are slow to greedify, but greedier updates risk instability via overestimation, creating a greedification-stability tradeoff.
Approach
Decouple the acting policy from the evaluated (critic) policy and apply a separate greedification operator to the evaluated policy (value improvement), keeping slow gradient-based improvement for the acting policy; prove convergence of the generalized policy iteration scheme for sufficient greedification operators.
Key results
- VI-TD3 and VI-SAC (expectile value improvement, tau=0.75) significantly outperform or match their baselines across the DeepMind control environments tested, across 20 seeds (10 for BoN).
- Increasing the expectile greedification parameter tau improves VI-TD3 performance up to tau=0.8, after which it degrades.
- Policy improvement alone is shown insufficient for convergence of exact policy iteration with stochastic policies.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add value improvement with expectile loss (τ=0.75) | TD3 and SAC · DeepMind control environments | performance | Mixed |
| Add value improvement using Best-of-N (N = 64) | TD3 · DeepMind control environments | performance | Helped |
| Increase value-improvement gradient steps on the same batch (ablation) | VI-TD3 · DeepMind continuous control environments | performance | Helped |
| Use expectile loss for implicit value improvement and increase τ (ablation) | VI-TD3 · DeepMind continuous control environments | performance | Mixed |
| Increase greedification of the evaluated policy with expectile loss beyond its best setting (ablation) | VI-TD3 · DeepMind continuous control environments | performance | Hurt |
| Increase gradient steps for the acting policy on the same batch (ablation) | VI-TD3 · DeepMind continuous control environments | sample efficiency | No clear effect |
| Increase replay ratio (ablation) | VI agent · DeepMind continuous control environments | performance | Mixed |
| Increase replay ratio versus spending the compute on value improvement (ablation) | VI agent · DeepMind continuous control environments | performance | Helped |
| Use value improvement rooted in a policy maintained during learning while varying its improvement rate (ablation) | VI agent · DeepMind continuous control environments | learning performance | Mixed |
Surprising findings
- Policy improvement alone is not sufficient for convergence of policy iteration with stochastic policies, even with exact evaluation.
- Repeating gradient steps on the same batch to greedify the evaluated policy did not effectively increase sample efficiency.
Limitations
- Theoretical convergence analysis restricted to finite-horizon, discrete state and finite action MDPs.
- Empirical evaluation limited to DeepMind control environments with TD3 and SAC.
- Gains from expectile operator may partly stem from overestimation increases in some environments.
Experimental setup
Datasets: DeepMind control suite
Benchmarks: DeepMind control suite
Metrics: episode return overestimation bias
Baselines: TD3 SAC BoN-based VI-TD3
Builds on: TD3 SAC Implicit Q-learning Gumbel MuZero Generalized Policy Iteration
Closest papers by meaning
- Learning Value Functions in Deep Policy Gradients using Residual Variance (ICLR 2021 · similarity 0.91)
- Greedy Actor-Critic: A New Conditional Cross-Entropy Method for Policy Improvement (ICLR 2023 · similarity 0.90)
- ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages (ICML 2024 · similarity 0.89)
- Wasserstein Policy Optimization (ICML 2025 · similarity 0.87)
- Actor-Free Continuous Control via Structurally Maximizable Q-Functions (NeurIPS 2025 · similarity 0.87)
- Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.