Research map / Multi-agent systems and games / Games and equilibria
Learning in Markov games
Proves convergence and hardness of decentralized policy learning in zero-sum and general-sum Markov games.
111 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Games and equilibria, keyword cluster “game · player · sum”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
28 papers in 2023–24 and 32 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 22.0% to 22.4%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Selectively Reactive Coordination for a Team of Robot Soccer Champions (AAAI 2016 · 24 citations)
- 2017 · Training Agent for First-Person Shooter Game with Actor-Critic Curriculum Learning (ICLR 2017 · 127 citations)
- 2018 · Multi-Agent Generative Adversarial Imitation Learning (NeurIPS 2018 · 124 citations)
- 2019 · Large Scale Learning of Agent Rationality in Two-Player Zero-Sum Games (AAAI 2019 · 18 citations)
- 2020 · Bi-Level Actor-Critic for Multi-Agent Coordination (AAAI 2020 · 77 citations)
- 2021 · Exploration-Exploitation in Multi-Agent Learning: Catastrophe Theory Meets Game Theory (AAAI 2021 · 38 citations)
- 2022 · Global Convergence of Multi-Agent Policy Gradient in Markov Potential Games (ICLR 2022 · 25 citations)
- 2023 · A Pair-Approximation Method for Modelling the Dynamics of Multi-Agent Stochastic Games (AAAI 2023 · 5 citations)
- 2024 · Accelerate Multi-Agent Reinforcement Learning in Zero-Sum Games with Subgame Curriculum Learning (AAAI 2024 · 8 citations)
- 2025 · Learning Nash Equilibrium of Markov Potential Games with a Shared Constraint via Primal-Dual Optimization (AAAI 2025 · 2 citations)
- 2026 · Provably Convergent Actor-Critic for MARL through Risk-aversion (ICML 2026 · 0 citations)
Earliest papers here
- Selectively Reactive Coordination for a Team of Robot Soccer Champions (AAAI 2016 · 24 citations)
- Training Agent for First-Person Shooter Game with Actor-Critic Curriculum Learning (ICLR 2017 · 127 citations)
- Countering Feedback Delays in Multi-Agent Learning (NeurIPS 2017 · 15 citations)
What has been tried
80 papers here report 484 measured changes (a component added, removed or swapped, compared with a baseline): helped 66%, no clear effect 7%, hurt 21%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add the full bias-plus-variance subgame sampling metric instead of bias-only sampling | MPE hard predator-prey · approximate exploitability | Helped | Accelerate Multi-Agent Reinforcement Learning in Zero-Sum Games with Subgame Curriculum Learning (AAAI 2024 · 8 citations) | |
| Use weighted output sampling with ρt = log(t + 1) instead of uniform output sampling | MPG with shared constraint · value function gap | O(ϵ + C²max/√T) -> O(ϵ + C²max/(√T log T)) | Helped | Learning Nash Equilibrium of Markov Potential Games with a Shared Constraint via Primal-Dual Optimization (AAAI 2025 · 2 citations) |
| remove the safety loss | Robot Warehouse · exploitability | 3.4 \times 10^{-2} versus 1.0 \times 10^{-3} | Helped | Convex Markov Games: A New Frontier for Multi-Agent Reinforcement Learning (ICML 2025 · 0 citations) |
| Remove self-play during training | Coin Game 3x3, against itself · average reward | Hurt | LOQA: Learning with Opponent Q-Learning Awareness (ICLR 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 23 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- P2MPO · 2 of 23 papers
- DR-NVI · 2 of 23 papers
- MMD · 2 of 23 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 60 papers; from OpenAlex and the papers' affiliations).
- Carnegie Mellon University · 6 papers
- Tsinghua University · 4 papers
- Massachusetts Institute of Technology · 4 papers
- Meta · 3 papers
- Peking University · 3 papers
- Google · 3 papers
- Imperial College London · 2 papers
- University of Maryland, College Park · 2 papers
- University of California, Berkeley · 2 papers
- University of Washington · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Cooperation and social dilemmas (95 papers)
- Opponent modeling in competitive games (75 papers)
- Mean-field game learning (21 papers)
Elsewhere in the topic
- Fair division and coalition formation (99 papers)
- Learning with strategic agents (74 papers)
- Strategic classification (51 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).