Research map / Reinforcement learning / Policy gradient and actor-critic
Temporal-difference and discounting theory
Analyze TD learning, discount factors, and value estimation error for prediction and control.
143 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Policy gradient and actor-critic, keyword cluster “value · td · distributional”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
31 papers in 2023–24 and 34 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 9.0% to 7.3%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Learning values across many orders of magnitude (NeurIPS 2016 · 86 citations)
- 2017 · Transfer Reinforcement Learning with Shared Dynamics (AAAI 2017 · 64 citations)
- 2018 · Meta-Gradient Reinforcement Learning (NeurIPS 2018 · 177 citations)
- 2019 · Information-Theoretic Considerations in Batch Reinforcement Learning (ICML 2019 · 96 citations)
- 2020 · Discovering Reinforcement Learning Algorithms (NeurIPS 2020 · 64 citations)
- 2021 · Reinforcement Learning with a Disentangled Universal Value Function for Item Recommendation (AAAI 2021 · 23 citations)
- 2022 · Efficient Continuous Control with Double Actors and Regularized Critics (AAAI 2022 · 58 citations)
- 2023 · Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learning (AAAI 2023 · 3 citations)
- 2024 · ERL-TD: Evolutionary Reinforcement Learning Enhanced with Truncated Variance and Distillation Mutation (AAAI 2024 · 6 citations)
- 2025 · The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret (ICML 2025 · 1 citation)
- 2026 · Rationality Measurement and Theory for Reinforcement Learning Agents (ICML 2026 · 0 citations)
Earliest papers here
- Learning values across many orders of magnitude (NeurIPS 2016 · 86 citations)
- Generalization and Exploration via Randomized Value Functions (ICML 2016 · 80 citations)
- Linear Feature Encoding for Reinforcement Learning (NeurIPS 2016 · 10 citations)
What has been tried
110 papers here report 972 measured changes (a component added, removed or swapped, compared with a baseline): helped 58%, no clear effect 12%, hurt 22%, mixed 9%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use distillation mutation instead of Gaussian mutation | Hopper-v2, Swimmer-v2, and Walker2d-v2 · cumulative reward | Helped | ERL-TD: Evolutionary Reinforcement Learning Enhanced with Truncated Variance and Distillation Mutation (AAAI 2024 · 6 citations) | |
| Remove the emphasis factor multiplying the TD error | Adaptive-emphasis prediction tasks · MSPBE | Hurt | Discerning Temporal Difference Learning (AAAI 2024 · 4 citations) | |
| Softmax parameterization of Q-values trained with MSE (no cross-entropy) | Atari online and offline RL · performance | No clear effect | Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (ICML 2024 · 3 citations) | |
| Add a fixed-point term to the PBO training loss | Chain-walk, K=2 and K=5 · L2 norm to optimal action-value function | Helped | Parameterized Projected Bellman Operator (AAAI 2024 · 2 citations) | |
| Increase lag time from τ=1 to τ≈30 | Mean first passage time and committor, n=40 · maximum relative asymptotic variance | τ≈30 | Helped | The surprising efficiency of temporal difference learning for rare event prediction (NeurIPS 2024 · 2 citations) |
| Increase critic ensemble size from 10 to 20 | MuJoCo tasks · performance | Helped | Symmetric Q-learning: Reducing Skewness of Bellman Error in Online Reinforcement Learning (AAAI 2024 · 1 citation) | |
| Add forward-in-time connections from upper to lower layers | MinAtar and DMC tasks · average episodic return | Helped | Temporal-Difference Learning Using Distributed Error Signals (NeurIPS 2024 · 1 citation) | |
| Use coarser state-feature approximations | Gridworld, distinct policies and distinct cumulants · MSE | Hurt | Adaptive Exploration for Data-Efficient General Value Function Evaluations (NeurIPS 2024 · 1 citation) | |
| Add the BYOL auxiliary loss | Humanoid · performance | Helped | Calibrated Value-Aware Model Learning with Probabilistic Environment Models (ICML 2025 · 0 citations) | |
| Use M = 2 instead of M = 1 | Inverted Pendulum · certified ROA volume | 42.9 ± 1.2 -> 76.7 ± 1.3 | Helped | Certifying Stability of Reinforcement Learning Policies using Generalized Lyapunov Functions (NeurIPS 2025 · 0 citations) |
| Use the derivative-based dTD prediction instead of the naive-dTD prediction in the β mixture | Hopper, HalfCheetah, Ant, and Humanoid · performance | Helped | A Temporal Difference Method for Stochastic Continuous Dynamics (NeurIPS 2025 · 0 citations) | |
| increase critic capacity | HumanoidBench-Medium · final performance | Helped | Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners (NeurIPS 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 20 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- SAC · 2 of 20 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 65 papers; from OpenAlex and the papers' affiliations).
- Google DeepMind · 4 papers
- University of Oxford · 3 papers
- University of California, Berkeley · 3 papers
- Politecnico di Milano · 2 papers
- The University of Tokyo · 2 papers
- University of Bristol · 2 papers
- University of Michigan - Ann Arbor · 2 papers
- King Abdullah University of Science and Technology · 2 papers
- University of Alberta · 2 papers
- University of Montreal · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Model-based RL with guarantees (245 papers)
- Regularized scalable deep value learning (216 papers)
- Tree search and policy optimization (109 papers)
- Distributional value learning (60 papers)
Elsewhere in the topic
- Safety layers and shielding (85 papers)
- Variance-reduced policy gradients (79 papers)
- Actor-critic theory (76 papers)
- Off-policy actor-critic with expressive policies (71 papers)
- Trajectory reuse in policy gradients (68 papers)
- Importance-weighted OPE estimators (57 papers)
- Alternative policy update rules (50 papers)
- Constrained policy optimization (48 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).