Research map / Reinforcement learning / MDPs and planning
Robust MDP policy gradient
Optimize policies in robust MDPs with rectangular uncertainty.
87 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in MDPs and planning, keyword cluster “markov decision · processes · observable”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
24 papers in 2023–24 and 31 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 11.8% to 17.7%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Safe Exploration in Finite Markov Decision Processes with Gaussian Processes (NeurIPS 2016 · 105 citations)
- 2017 · Hindsight Optimization for Hybrid State and Action MDPs (AAAI 2017 · 13 citations)
- 2018 · Safe Exploration and Optimization of Constrained MDPs Using Gaussian Processes (AAAI 2018 · 93 citations)
- 2019 · Real-Time Reinforcement Learning (NeurIPS 2019 · 14 citations)
- 2020 · Reinforcement Learning with Non-Markovian Rewards (AAAI 2020 · 54 citations)
- 2021 · Robust Reinforcement Learning using Least Squares Policy Iteration with Provable Performance Guarantees (ICML 2021 · 15 citations)
- 2022 · Policy Learning for Robust Markov Decision Process with a Mismatched Generative Model (AAAI 2022 · 4 citations)
- 2023 · Robust Average-Reward Markov Decision Processes (AAAI 2023 · 6 citations)
- 2024 · Solving Non-rectangular Reward-Robust MDPs via Frequency Regularization (AAAI 2024 · 2 citations)
- 2025 · How to Find the Exact Pareto Front for Multi-Objective MDPs? (ICLR 2025 · 1 citation)
- 2026 · Dynamic Programming for Epistemic Uncertainty in Markov Decision Processes (ICML 2026 · 0 citations)
Earliest papers here
- Safe Exploration in Finite Markov Decision Processes with Gaussian Processes (NeurIPS 2016 · 105 citations)
- Exploiting Anonymity in Approximate Linear Programming: Scaling to Large Multiagent MDPs (AAAI 2016 · 24 citations)
- Hindsight Optimization for Hybrid State and Action MDPs (AAAI 2017 · 13 citations)
What has been tried
60 papers here report 384 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 7%, hurt 13%, mixed 8%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| use coupled (non-rectangular) L2 reward uncertainty set instead of s-rectangular set | Tabular RMDP with Gaussian test rewards · CVaR of worst 5% returns | Helped | Solving Non-rectangular Reward-Robust MDPs via Frequency Regularization (AAAI 2024 · 2 citations) | |
| Remove states with empty language from the complement | 10,000 randomly generated formulas (mean) · states | 48,367.45 -> 3,748.94 | Helped | Omega-Regular Decision Processes (AAAI 2024 · 1 citation) |
| Increase memory allocated to vertex R | Fig. 1 graph · L-badness | Helped | Optimizing Local Satisfaction of Long-Run Average Objectives in Markov Decision Processes (AAAI 2024 · 1 citation) | |
| Decrease kappa | DMC walker, noise perturbation · performance under perturbation | Helped | Bring Your Own (Non-Robust) Algorithm to Solve Robust MDPs by Estimating The Worst Kernel (ICML 2024 · 1 citation) | |
| remove pessimism for unobserved rewards | Bottleneck, solvable · test return | No clear effect | Model-Based Exploration in Monitored Markov Decision Processes (ICML 2025 · 0 citations) | |
| Replace KL regularization with ℓ2 regularization (RPPG) | Garnet · reward | Hurt | Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guarantees (NeurIPS 2025 · 0 citations) | |
| Train with 10% of the training set instead of the full set | Retro*-190 · direct-generation success rate | 92% | Mixed | Retrosynthesis Planning via Worst-path Policy Optimisation in Tree-structured MDPs (NeurIPS 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 18 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- Garnet · 2 of 18 papers
Baselines
- value iteration · 2 of 18 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 55 papers; from OpenAlex and the papers' affiliations).
- Technion - Israel Institute of Technology · 5 papers
- University of New Hampshire · 3 papers
- City University of Hong Kong · 3 papers
- Air Force Research Laboratory · 2 papers
- University of Colorado Boulder · 2 papers
- Harvard University · 2 papers
- Duke University · 2 papers
- Cornell University · 2 papers
- University of Wisconsin - Madison · 2 papers
- Google · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Average-reward and regularized MDPs (135 papers)
- POMDPs and confounded RL (99 papers)
- POMDP policy optimization (68 papers)
Elsewhere in the topic
- Function approximation in RL theory (93 papers)
- Offline RL sample complexity (86 papers)
- Policy optimization theory (61 papers)
- Rich-observation exploration (53 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).