Research map / Reinforcement learning / Policy gradient and actor-critic
Alternative policy update rules
Replace standard gradient steps with evolved, natural-gradient, or divide-and-conquer updates.
50 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Policy gradient and actor-critic, keyword cluster “policy gradient · gradients · variance”
State History Tried Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
5 papers in 2023–24 and 13 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 1.5% to 2.8%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Self-Critical Sequence Training for Image Captioning (CVPR 2017 · 2,268 citations)
- 2018 · Evolved Policy Gradients (NeurIPS 2018 · 133 citations)
- 2019 · CEM-RL: Combining evolutionary and gradient-based methods for policy search (ICLR 2019 · 93 citations)
- 2020 · DISK: Learning local features with policy gradient (NeurIPS 2020 · 192 citations)
- 2021 · Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online (ICLR 2021 · 2 citations)
- 2022 · TTOpt: A Maximum Volume Quantized Tensor Train-based Optimization and its Application to Reinforcement Learning (NeurIPS 2022 · 9 citations)
- 2023 · Quantum Policy Gradient Algorithm with Optimized Action Decoding (ICML 2023 · 6 citations)
- 2024 · DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning (NeurIPS 2024 · 3 citations)
- 2025 · Gradient Boosting Reinforcement Learning (ICML 2025 · 1 citation)
- 2026 · Credit-assigned Policy Gradient for Early Stage Retrieval in Two-stage Ranking (ICML 2026 · 0 citations)
Earliest papers here
- Self-Critical Sequence Training for Image Captioning (CVPR 2017 · 2,268 citations)
- Batch Policy Gradient Methods for Improving Neural Conversation Models (ICLR 2017 · 7 citations)
- Evolved Policy Gradients (NeurIPS 2018 · 133 citations)
What has been tried
39 papers here report 442 measured changes (a component added, removed or swapped, compared with a baseline): helped 60%, no clear effect 12%, hurt 21%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add policy-gradient loss and backpropagate it through trajectory optimization | DeepMind Control Suite, 15 tasks · return | Helped | DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning (NeurIPS 2024 · 3 citations) | |
| Vary the policy or value learning rate | CartPole-v1 · average episodic reward | Mixed | Gradient Boosting Reinforcement Learning (ICML 2025 · 1 citation) | |
| use risk-seeking quantile α=0.1 rather than α=0.5 | 3-D trig dynamics · success rate | 33.33% -> 66.67% | Helped | Analytical Lyapunov Function Discovery: An RL-based Generative Approach (ICML 2025 · 0 citations) |
| Increase encoder skips from 1 to 5 residual blocks | Atari-10 · aggregate mean HNS | 0.25 -> 0.58 | Helped | Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning (NeurIPS 2025 · 0 citations) |
| Add a BC term to the method with jump-start | Crazyflie control · average final reward | 334 ± 25.59 -> 412 ± 6.72 | Helped | Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks (NeurIPS 2025 · 0 citations) |
| Vary λ in RL-U; values other than λ=1 | 256×256 hyper-grid · D_TV convergence speed | Helped | GFlowNet Training by Policy Gradients (ICML 2024 · 0 citations) |
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 18 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 2 papers
- Mila · 2 papers
- Google · 2 papers
- Université de Montréal · 2 papers
- Harvard University · 2 papers
- Delft University of Technology · 2 papers
Use it
Download BibTeX of 36 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Variance-reduced policy gradients (79 papers)
- Trajectory reuse in policy gradients (68 papers)
- Policy gradients for special settings (45 papers)
Elsewhere in the topic
- Model-based RL with guarantees (245 papers)
- Regularized scalable deep value learning (216 papers)
- Temporal-difference and discounting theory (143 papers)
- Tree search and policy optimization (109 papers)
- Safety layers and shielding (85 papers)
- Actor-critic theory (76 papers)
- Off-policy actor-critic with expressive policies (71 papers)
- Distributional value learning (60 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).