Research map / Reinforcement learning / MDPs and planning
Average-reward and regularized MDPs
Develop theory for regularized, average-reward, and non-stationary MDPs.
135 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in MDPs and planning, keyword cluster “markov decision · processes · observable”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
29 papers in 2023–24 and 39 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 14.3% to 22.3%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Reinforcement Learning with Parameterized Actions (AAAI 2016 · 166 citations)
- 2017 · Optimizing Quantiles in Preference-Based Markov Decision Processes (AAAI 2017 · 7 citations)
- 2018 · Planning and Learning for Decentralized MDPs With Event Driven Rewards (AAAI 2018 · 9 citations)
- 2019 · A Theory of Regularized Markov Decision Processes (ICML 2019 · 116 citations)
- 2020 · Scalable Methods for Computing State Similarity in Deterministic Markov Decision Processes (AAAI 2020 · 66 citations)
- 2021 · Lipschitz Lifelong Reinforcement Learning (AAAI 2021 · 21 citations)
- 2022 · Learning to Branch with Tree MDPs (NeurIPS 2022 · 17 citations)
- 2023 · BQ-NCO: Bisimulation Quotienting for Efficient Neural Combinatorial Optimization (NeurIPS 2023 · 10 citations)
- 2024 · Principal-Agent Reward Shaping in MDPs (AAAI 2024 · 5 citations)
- 2025 · Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs (AAAI 2025 · 2 citations)
- 2026 · Learning the Minimum Action Distance (ICML 2026 · 0 citations)
Earliest papers here
- Reinforcement Learning with Parameterized Actions (AAAI 2016 · 166 citations)
- Truncated Approximate Dynamic Programming with Task-Dependent Terminal Value (AAAI 2016 · 4 citations)
- Optimizing Quantiles in Preference-Based Markov Decision Processes (AAAI 2017 · 7 citations)
What has been tried
101 papers here report 707 measured changes (a component added, removed or swapped, compared with a baseline): helped 63%, no clear effect 8%, hurt 21%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use algorithm features in the metalevel state | Deep Sea Treasure (DST) · normalised total thinking and acting cost | Helped | Stop! Planner Time: Metareasoning for Probabilistic Planning Using Learned Performance Profiles (AAAI 2024 · 3 citations) | |
| Add Q-function loss regularization toward the empirical-MDP value function | Breakout and HalfCheetah · mean score | Helped | Exploiting the Replay Memory Before Exploring the Environment: Enhancing Reinforcement Learning Through Empirical MDP Iteration (NeurIPS 2024 · 2 citations) | |
| add separate reward predictors conditioned on termination rather than one unified predictor | Catch Point · return | -17.65±7.75 -> 7.56±4.86 | Helped | Model-based Reinforcement Learning for Parameterized Action Spaces (ICML 2024 · 1 citation) |
| Add an ensemble of critics to DecQN | Humanoid-Stand · return | 604.82 ± 85.1 -> 832.99 ± 11.3 | Helped | REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processes (ICLR 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 24 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- MDP-OOMD · 2 of 24 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 68 papers; from OpenAlex and the papers' affiliations).
- Purdue University · 4 papers
- Harvard University · 3 papers
- University of Oxford · 3 papers
- The Chinese University of Hong Kong · 3 papers
- Peking University · 3 papers
- Tsinghua University · 2 papers
- Google DeepMind · 2 papers
- Technische Universität München · 2 papers
- King's College London · 2 papers
- Tel Aviv University · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- POMDPs and confounded RL (99 papers)
- Robust MDP policy gradient (87 papers)
- POMDP policy optimization (68 papers)
Elsewhere in the topic
- Function approximation in RL theory (93 papers)
- Offline RL sample complexity (86 papers)
- Policy optimization theory (61 papers)
- Rich-observation exploration (53 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).