Research map / Reinforcement learning / Policy gradient and actor-critic
Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically combines over 60 SAC regularization variants (critic, network, plasticity) across 14 DMC and MetaWorld tasks at two replay ratios, finding network regularization (layer norm, spectral norm) and resets outperform critic regularization such as clipped double Q-learning.
Problem
Off-policy actor-critic methods use many regularization techniques at high replay ratios, but these have been tested in narrow settings and against few baselines, so the mechanisms behind their gains are unclear.
Approach
Factorially combine 12 SAC design choices from three groups (critic, network, plasticity regularization) and analyze marginalized performance together with proxies for overestimation, overfitting, and plasticity loss, across many tasks and two benchmarks.
Key results
- Network regularization and plasticity regularization (especially resets) are generally more robust than critic regularization; removing critic regularization (!CR) helps on DMC and MetaWorld.
- Layer norm combined with resets at RR=16 largely solves Dog-Run and Dog-Trot, reaching performance close to a model-based agent on proprioceptive states.
- Overestimation shows the strongest Spearman correlation with return across benchmarks, and layer/spectral norm reduce approximation error more than critic regularizers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add layer normalization to the critic (ablation) | DrQ · Acrobot Swingup (image-based) | final performance: 172.9\pm22.1 -> 87.6 \pm 19.7 | Hurt |
| Add full-parameter resets every 200k environment steps (ablation) | DrQ · Acrobot Swingup (image-based) | final performance: 172.9\pm22.1 -> 52.3 \pm 11.2 | Hurt |
| Add critic layer normalization and full-parameter resets every 200k environment steps (ablation) | DrQ · Acrobot Swingup (image-based) | final performance: 172.9\pm22.1 -> 88.8 \pm 8.2 | Hurt |
| Add layer normalization to the critic (ablation) | DrQ · Cheetah Run (image-based) | final performance: 727.4\pm7.3 -> 680.0 \pm 2.3 | Hurt |
| Add full-parameter resets every 200k environment steps (ablation) | DrQ · Cheetah Run (image-based) | final performance: 727.4\pm7.3 -> 715.6 \pm 7.1 | Hurt |
| Add critic layer normalization and full-parameter resets every 200k environment steps (ablation) | DrQ · Cheetah Run (image-based) | final performance: 727.4\pm7.3 -> 653.9 \pm 2.1 | Hurt |
| Add layer normalization to the critic (ablation) | DrQ · Hopper Hop (image-based) | final performance: 74.6\pm34.9 -> 116.3 \pm 27.2 | Helped |
| Add full-parameter resets every 200k environment steps (ablation) | DrQ · Hopper Hop (image-based) | final performance: 74.6\pm34.9 -> 135.7 \pm 24.1 | Helped |
| Add critic layer normalization and full-parameter resets every 200k environment steps (ablation) | DrQ · Hopper Hop (image-based) | final performance: 74.6\pm34.9 -> 167.3 \pm 14.3 | Helped |
| Add layer normalization to the critic (ablation) | DrQ · Humanoid Stand (image-based) | final performance: 7.8\pm0.2 -> 8.1 \pm 0.2 | Helped |
| Add full-parameter resets every 200k environment steps (ablation) | DrQ · Humanoid Stand (image-based) | final performance: 7.8\pm0.2 -> 7.5 \pm 0.4 | Hurt |
| Add critic layer normalization and full-parameter resets every 200k environment steps (ablation) | DrQ · Humanoid Stand (image-based) | final performance: 7.8\pm0.2 -> 7.8 \pm 0.4 | No clear effect |
Surprising findings
- Clipped Double Q-learning significantly degrades performance on MetaWorld manipulation tasks and Hopper Hop, even when paired with other regularizers.
- Image-based DrQ variants with layer norm or resets did not improve over vanilla DrQ, so proprioceptive gains did not transfer directly to image-based control.
Limitations
- Evaluation limited to proprioceptive tasks on DMC and MetaWorld and to the SAC method.
- Hyperparameters taken from original papers of each intervention, not tuned per combination.
- Correlation analyses are observational and environment-dependent; overfitting coefficient signs vary across environments.
Experimental setup
Datasets: DeepMind Control Suite MetaWorld
Benchmarks: DeepMind Control Suite MetaWorld
Metrics: IQM return overestimation overfitting ratio dormant neurons representation rank gradient norm parameter norm Spearman correlation
Baselines: SAC CDQ TOP GPL CReLU SAM ReDo DrQ
Builds on: Soft Actor-Critic Clipped Double Q-learning Tactical Optimism and Pessimism Generalized Pessimism Learning Sharpness-Aware Minimization
Closest papers by meaning
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.90)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.90)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.90)
- Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.