Research map / Reinforcement learning / Policy gradient and actor-critic
Stabilizing deep actor-critic
Study implementation choices, overestimation, and normalization to make deep actor-critic reliable.
48 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Policy gradient and actor-critic, keyword cluster “actor critic · policy actor · critic algorithms”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
15 papers in 2023–24 and 12 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 4.4% to 2.6%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning (ICLR 2017 · 29 citations)
- 2018 · Action Branching Architectures for Deep Reinforcement Learning (AAAI 2018 · 247 citations)
- 2019 · Self-Critical N-Step Training for Image Captioning (CVPR 2019 · 63 citations)
- 2020 · Trust the Model When It Is Confident: Masked Model-based Actor-Critic (NeurIPS 2020 · 21 citations)
- 2021 · What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study (ICLR 2021 · 47 citations)
- 2022 · Dropout Q-Functions for Doubly Efficient Reinforcement Learning (ICLR 2022 · 19 citations)
- 2023 · Decision S4: Efficient Sequence-Based RL via State Spaces Layers (ICLR 2023 · 3 citations)
- 2024 · Two-Stage Evolutionary Reinforcement Learning for Enhancing Exploration and Exploitation (AAAI 2024 · 9 citations)
- 2025 · A Model of Place Field Reorganization During Reward Maximization (ICML 2025 · 3 citations)
- 2026 · This State Looks Like That: Self-Interpretable Reinforcement Learning Agents using Prototype Soft Actor-Critic (ICML 2026 · 0 citations)
Earliest papers here
- Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning (ICLR 2017 · 29 citations)
- Action Branching Architectures for Deep Reinforcement Learning (AAAI 2018 · 247 citations)
- Learning With Options That Terminate Off-Policy (AAAI 2018 · 13 citations)
What has been tried
38 papers here report 398 measured changes (a component added, removed or swapped, compared with a baseline): helped 63%, no clear effect 10%, hurt 22%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| remove the exploration stage | Hopper, Ant, and Walker2d · episode return | Hurt | Two-Stage Evolutionary Reinforcement Learning for Enhancing Exploration and Exploitation (AAAI 2024 · 9 citations) | |
| Add the transformer-based encoder | Sim1 · F1 | 0.55±0.13 -> 0.63 ± 0.17 | Helped | MetaRLEC: Meta-Reinforcement Learning for Discovery of Brain Effective Connectivity (AAAI 2024 · 7 citations) |
| Optimize field widths | Single fixed target · cumulative discounted reward (G) | Helped | A Model of Place Field Reorganization During Reward Maximization (ICML 2025 · 3 citations) | |
| Add layer normalization to the critic | Acrobot Swingup (image-based) · final performance | 172.9±22.1 -> 87.6 ± 19.7 | Hurt | Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning (ICML 2024 · 3 citations) |
| use pink noise exploration instead of deterministic or white noise exploration | Humanoid · performance and learning stability | Helped | Trust the Model Where It Trusts Itself - Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption (ICML 2024 · 2 citations) | |
| replace cross-entropy with ordinal cross-entropy | Hopper, HalfCheetah, Walker2d, and Reacher under GCM perturbations · episodic return | 2198.8 -> 2677.5; 851.5 -> 2127.3; 1747.6 -> 2058.6; -12.2 -> -7.8 | Helped | The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards (AAAI 2025 · 1 citation) |
| Add online Q-function updates | Swimmer · evaluation reward | Helped | Coprocessor Actor Critic: A Model-Based Reinforcement Learning Approach For Adaptive Brain Stimulation (ICML 2024 · 1 citation) | |
| Add expert trajectory regularization to the DRL policy | AF dataset test, 2-step · exact prediction (= 0) | 0.029 -> 0.439 | Helped | Stabilizing and Accelerating Autofocus with Expert Trajectory Regularized Deep Reinforcement Learning (CVPR 2025 · 0 citations) |
| Use 10 or 20 diffusion steps rather than 30 | Ant-v4 · performance | Helped | Efficient Online Reinforcement Learning for Diffusion Policy (ICML 2025 · 0 citations) | |
| anneal τ from 0.9 rather than fix τ at 0.7 | DM Control, 3M steps · Mean | 730.7 -> 746.1 | Helped | Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning (ICML 2025 · 0 citations) |
| Add the noisy label alignment reward | CIFAR100-IDN, noise 0.50 · test accuracy (%) | 72.0 -> 74.7 | Helped | Learning to Clean: Reinforcement Learning for Noisy Label Correction (NeurIPS 2025 · 0 citations) |
| Use standard SAC exploration instead of ε-greedy exploration | 200 million training regime · performance | Helped | CALE: Continuous Arcade Learning Environment (NeurIPS 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 8 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- PPO · 3 of 8 papers
- SAC · 3 of 8 papers
- TD3 · 2 of 8 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 27 papers; from OpenAlex and the papers' affiliations).
- Google · 3 papers
- University of California, Berkeley · 2 papers
- Harvard University · 2 papers
Use it
Download BibTeX of 38 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Actor-critic theory (76 papers)
- Off-policy actor-critic with expressive policies (71 papers)
Elsewhere in the topic
- Model-based RL with guarantees (245 papers)
- Regularized scalable deep value learning (216 papers)
- Temporal-difference and discounting theory (143 papers)
- Tree search and policy optimization (109 papers)
- Safety layers and shielding (85 papers)
- Variance-reduced policy gradients (79 papers)
- Trajectory reuse in policy gradients (68 papers)
- Distributional value learning (60 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).