Research map / Reinforcement learning / Policy gradient and actor-critic
Off-policy actor-critic with expressive policies
Improve off-policy actor-critic by exploiting past success, stochastic policies, and value-policy links.
71 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Policy gradient and actor-critic, keyword cluster “actor critic · policy actor · critic algorithms”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
12 papers in 2023–24 and 20 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 3.5% to 4.3%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Asynchronous Methods for Deep Reinforcement Learning (ICML 2016 · 1,685 citations)
- 2017 · Reinforcement Learning with Deep Energy-Based Policies (ICML 2017 · 409 citations)
- 2018 · Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (ICML 2018 · 3,754 citations)
- 2019 · Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction (NeurIPS 2019 · 351 citations)
- 2020 · Discretizing Continuous Action Space for On-Policy Optimization (AAAI 2020 · 85 citations)
- 2021 · Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (ICLR 2021 · 126 citations)
- 2022 · Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning (ICLR 2022 · 63 citations)
- 2023 · Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation (NeurIPS 2023 · 3 citations)
- 2024 · Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic (ICML 2024 · 3 citations)
- 2025 · Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation (ICML 2025 · 0 citations)
- 2026 · PAC-Bayesian Reinforcement Learning Trains Generalizable Policies (ICML 2026 · 0 citations)
Earliest papers here
- Asynchronous Methods for Deep Reinforcement Learning (ICML 2016 · 1,685 citations)
- Continuous Deep Q-Learning with Model-based Acceleration (ICML 2016 · 320 citations)
- Reinforcement Learning with Deep Energy-Based Policies (ICML 2017 · 409 citations)
What has been tried
56 papers here report 563 measured changes (a component added, removed or swapped, compared with a baseline): helped 61%, no clear effect 9%, hurt 21%, mixed 9%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use BAC instead of SAC | DKittyWalk-Medium · recovery speed after adding expert trajectories | Helped | Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic (ICML 2024 · 3 citations) | |
| Set new actors per iteration to N=3 | HalfCheetah-v2 · average return | Helped | Double Buffers CEM-TD3: More Efficient Evolution and Richer Exploration (AAAI 2024 · 2 citations) | |
| Use parameterized Gaussian initialization for the SVGD sampler | MuJoCo benchmark · return | Helped | S$2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic (ICLR 2024 · 2 citations) | |
| remove the gradient-dormancy-guided reset mechanism | MetaWorld dense and sparse reward tasks · performance | Hurt | ACE: Off-Policy Actor-Critic with Causality-Aware Entropy Regularization (ICML 2024 · 2 citations) | |
| Replace the dual-critic structure with a standard critic only | DistShift2, LavaCrossingS9N2, LavaGapS6, MultiRoom-N2-S4 · return | Hurt | Dual Critic Reinforcement Learning under Partial Observability (NeurIPS 2024 · 2 citations) | |
| include policy entropy when computing the Q-function | Mujoco benchmarks · performance | Helped | Maximum Entropy Reinforcement Learning with Diffusion Policy (ICML 2025 · 0 citations) | |
| Add input L2 normalization | All (57), design study · normalized performance | 0.809 ± 0.059 -> 0.892 ± 0.032 | Helped | Hyperspherical Normalization for Scalable Deep Reinforcement Learning (ICML 2025 · 0 citations) |
| Add Successful Transition Replay (STR) | AndroidWorld 80-task subset · success rate | Helped | Succeed or Learn Slowly: Sample Efficient Off-Policy Reinforcement Learning for Mobile App Control (NeurIPS 2025 · 0 citations) | |
| add conditional Q-value generation | Hopper-v4, BipedalWalker-v3, Walker2d-v4, HalfCheetah-v4 · return | Q3C: 3206.14 ± 407.23, 290.11 ± 26.43, 3977.39 ± 879.70, 9468.66 ± 949.01; without: 2329.9 ± 610.1, 286.0 ± 46.7, 3614.3 | Helped | Actor-Free Continuous Control via Structurally Maximizable Q-Functions (NeurIPS 2025 · 0 citations) |
| Add an adaptive horizon objective | Ant · asymptotic episodic reward | Helped | Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulation (ICML 2024 · 0 citations) | |
| remove the distribution discriminator | stationary and domain adaptation environments · learning performance | Hurt | OMPO: A Unified Framework for RL under Policy and Dynamics Shifts (ICML 2024 · 0 citations) | |
| remove Delayed f(Q) Update | Ant · total_return | Hurt | RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learning (ICML 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 8 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- SAC · 4 of 8 papers
- PPO · 3 of 8 papers
- TD3 · 2 of 8 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 32 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 4 papers
- Meta · 2 papers
- NVIDIA · 2 papers
- Universität Hamburg · 2 papers
- Chinese Academy of Sciences · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Actor-critic theory (76 papers)
- Stabilizing deep actor-critic (48 papers)
Elsewhere in the topic
- Model-based RL with guarantees (245 papers)
- Regularized scalable deep value learning (216 papers)
- Temporal-difference and discounting theory (143 papers)
- Tree search and policy optimization (109 papers)
- Safety layers and shielding (85 papers)
- Variance-reduced policy gradients (79 papers)
- Trajectory reuse in policy gradients (68 papers)
- Distributional value learning (60 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).