Research map / Reinforcement learning / Policy gradient and actor-critic
Tree search and policy optimization
Connect Monte-Carlo tree search and sequential Monte Carlo to regularized policy optimization.
109 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Policy gradient and actor-critic, keyword cluster “value · td · distributional”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
29 papers in 2023–24 and 28 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 8.5% to 6.0%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Reliable Decision Support using Counterfactual Models (NeurIPS 2017 · 86 citations)
- 2018 · Multi-Step Reinforcement Learning: A Unifying Algorithm (AAAI 2018 · 111 citations)
- 2019 · Maximum Entropy-Regularized Multi-Goal Reinforcement Learning (ICML 2019 · 48 citations)
- 2020 · Monte-Carlo Tree Search as Regularized Policy Optimization (ICML 2020 · 38 citations)
- 2021 · Universal Trading for Order Execution with Oracle Policy Distillation (AAAI 2021 · 33 citations)
- 2022 · Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer (ICML 2022 · 4 citations)
- 2023 · Off-Policy Proximal Policy Optimization (AAAI 2023 · 15 citations)
- 2024 · A Reinforcement-Learning-Based Multiple-Column Selection Strategy for Column Generation (AAAI 2024 · 7 citations)
- 2025 · Optimizing Vital Sign Monitoring in Resource-Constrained Maternal Care: An RL-Based Restless Bandit Approach (AAAI 2025 · 1 citation)
- 2026 · Stability-Aware Reinforcement Learning for Robust Class Integration Test Order Generation (AAAI 2026 · 1 citation)
Earliest papers here
- Reliable Decision Support using Counterfactual Models (NeurIPS 2017 · 86 citations)
- Sequential Classification-Based Optimization for Direct Policy Search (AAAI 2017 · 53 citations)
- Consistent On-Line Off-Policy Evaluation (ICML 2017 · 32 citations)
What has been tried
82 papers here report 707 measured changes (a component added, removed or swapped, compared with a baseline): helped 62%, no clear effect 9%, hurt 22%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| remove complete-graph embeddings | CSP Easy · # Itr | 10.91 -> 10.33 | Hurt | A Reinforcement-Learning-Based Multiple-Column Selection Strategy for Column Generation (AAAI 2024 · 7 citations) |
| use quantile distribution instead of Gaussian distribution for the policy return distribution | HalfCheetah-v2 · average return | 14796.2±1473.2 -> 16484.3±1373.75 | Helped | OVD-Explorer: Optimism Should Not Be the Sole Pursuit of Exploration in Noisy Environments (AAAI 2024 · 7 citations) |
| Use intermediate λ values rather than λ = 0 | Random chain domain · MSTDE | Helped | From Past to Future: Rethinking Eligibility Traces (AAAI 2024 · 3 citations) | |
| Increase MC node count from m=5 to m=10 | (T, Q̄) = (10, 10), (30, 30), and (100, 100) · average test rewards | 21.77 -> 21.80; 56.71 -> 62.84; 115.88 -> 132.62 | Helped | Optimal Mechanism in a Dynamic Stochastic Knapsack Environment (AAAI 2024 · 2 citations) |
| increase initial scale from σinit = 0.5 to σinit = 1.0 | MuJoCo 8 tasks · normalization score | 0.747 -> 0.812 | Helped | Truncated Gaussian Policy for Debiased Continuous Control (AAAI 2025 · 1 citation) |
| maximize NSW rather than utilitarian welfare | MO-Four-Room · NSW | -17.33 -> -10.77 | Helped | FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learning (NeurIPS 2025 · 1 citation) |
| Increase risk-aversion to β=2.0 | Adult Income (n=2000) · success rate | 0.995 -> 0.717 | Hurt | SafeAR: Safe Algorithmic Recourse by Risk-Aware Policies (AAAI 2024 · 1 citation) |
| Use a 10K-step pretraining budget plus 2.5K fine-tuning steps rather than the 50K-step DAD baseline | Source Location Finding · total EIG | Helped | Step-DAD: Semi-Amortized Policy-Based Bayesian Experimental Design (ICML 2025 · 0 citations) | |
| use a constrained, Q-twisted proposal with α = 0.1 instead of the prior proposal with α = 0 | Jumanji environments at low particle budgets · final test performance | Helped | Trust-Region Twisted Policy Improvement (ICML 2025 · 0 citations) | |
| increase capacity dependence from gamma=0 to gamma=0.5 | FO, capacity-dependent policy · speedup | 441x -> 427x | Hurt | Speeding up Policy Simulation in Supply Chain RL (ICML 2025 · 0 citations) |
| Add backward KL regularization and trajectory augmentation to OP-TB | NATS-Bench · sampling efficiency (accuracy vs T&T time) | Helped | Order-Preserving GFlowNets (ICLR 2024 · 0 citations) | |
| Increase the bias coefficient $\alpha$ | Attention allocation · bias | Helped | Adapting Static Fairness to Sequential Decision-Making: Bias Mitigation Strategies towards Equal Long-term Benefit Rate (ICML 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 17 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- D4MORL · 2 of 17 papers
Baselines
- Random · 2 of 17 papers
- PPO · 2 of 17 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 57 papers; from OpenAlex and the papers' affiliations).
- Massachusetts Institute of Technology · 4 papers
- Tsinghua University · 3 papers
- Korea University · 3 papers
- Seoul National University · 3 papers
- Harvard University · 3 papers
- University of Maryland, College Park · 3 papers
- Carnegie Mellon University · 2 papers
- University of California, Berkeley · 2 papers
- J.P. Morgan AI Research · 2 papers
- Google DeepMind · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Model-based RL with guarantees (245 papers)
- Regularized scalable deep value learning (216 papers)
- Temporal-difference and discounting theory (143 papers)
- Distributional value learning (60 papers)
Elsewhere in the topic
- Safety layers and shielding (85 papers)
- Variance-reduced policy gradients (79 papers)
- Actor-critic theory (76 papers)
- Off-policy actor-critic with expressive policies (71 papers)
- Trajectory reuse in policy gradients (68 papers)
- Importance-weighted OPE estimators (57 papers)
- Alternative policy update rules (50 papers)
- Constrained policy optimization (48 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).