Research map / Reinforcement learning / Policy gradient and actor-critic
A Self-Tuning Actor-Critic Algorithm
Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado P van Hasselt, David Silver, Satinder P. Singh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add online metagradient self-tuning of differentiable actor-critic loss parameters | IMPALA on Atari · Atari (57 games, 200M frames) | median human-normalized score: 243% -> 364% | Improved |
| Add online metagradient self-tuning | IMPALA on DM Control with features · DM Control, features (30M steps) | mean score: 217 to 389 | Improved |
| Add online metagradient self-tuning | IMPALA on DM Control with pixels · DM Control, pixels (30M steps) | mean score: 108 to 202 | Improved |
| Add online metagradient self-tuning | IMPALA on RWRL · RWRL (30M steps) | mean score: 195 to 295 | Improved |
| Add auxiliary self-tuned actor-critic tasks | STAC on Atari · Atari | median human-normalized score: null | Improved |
| Self-tune more metaparameters (ablation) | STAC and STACX on Atari · Atari | median human-normalized score | Improved |
| Add γ embedding while self-tuning γ and λ (ablation) | STACX on Atari · Atari | score: 280.6% | Improved |
| Add γ embedding to STACX with all metaparameters self-tuned (ablation) | STACX on Atari · Atari | score: 306.8%, 307.7% versus 364% | Hurt |
| Learn both αρ and αc without enforcing αρ ≥ αc (ablation) | STACX on Atari · Atari (57 games) | median human-normalized score: 353% | Hurt |
| Self-tune α together with a single truncation parameter ρ̄ = c̄ (ablation) | STACX on Atari · Atari (57 games) | median human-normalized score: 301% | Hurt |
| Use a random ensemble or average-logit ensemble so auxiliary heads can act (ablation) | STACX · Experiments reported in the paper | performance | Hurt |
| Use 2, 5, or 8 auxiliary losses instead of 3 (ablation) | STACX on Atari · Atari | performance | Mixed |
Closest papers by meaning
- SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer (ICML 2026 · similarity 0.89)
- Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning (ICML 2024 · similarity 0.87)
- Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning (ICML 2024 · similarity 0.87)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.87)
- Offline Actor-Critic Reinforcement Learning Scales to Large Models (ICML 2024 · similarity 0.87)
- Actor-Critics Can Achieve Optimal Sample Efficiency (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
