Research map / Optimization / Nonconvex and smooth optimization
Policy Optimization for Markov Games: Unified Framework and Faster Convergence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OFTRL with smooth value updates instead of standard FTRL-style updates | Policy optimization for two-player zero-sum Markov games · Two-player zero-sum Markov games, full-information | NEGap convergence rate: \widetilde{\mathcal{O}}(T^{-1/2}) -> \widetilde{\mathcal{O}}(T^{-5/6}) | Helped |
| Replace the shared value estimate with separate upper and lower value estimates (ablation) | OFTRL for two-player zero-sum Markov games · Two-player zero-sum Markov games, full-information | NEGap convergence rate: \widetilde{\mathcal{O}}(T^{-5/6}) -> \widetilde{\mathcal{O}}(T^{-1}) | Helped |
| Use OFTRL for policy updates in place of a V-learning-style baseline | Policy optimization for multi-player general-sum Markov games · Multi-player general-sum Markov games, full-information | CCE convergence rate: \widetilde{\mathcal{O}}(T^{-1/2}) -> \widetilde{\mathcal{O}}(T^{-3/4}) | Helped |
| Use smooth rather than eager value updates (ablation) | Policy optimization for two-player zero-sum Markov games · Two-layer zero-sum Markov game (H=2, S=4, A=2) | NEGap convergence: T^{-.308} (INPG, η=T^{-1/2}) versus T^{-.570} (FTRL) | Helped |
| Use eager value updates with INPG and η=1 (ablation) | INPG · Two-layer zero-sum Markov game (H=2, S=4, A=2) | NEGap convergence | Hurt |
| Increase OFTRL's learning rate from η=T^{-1/6} to η=1 (ablation) | OFTRL with smooth value updates · Two-layer zero-sum Markov game (H=2, S=4, A=2) | NEGap convergence exponent: T^{-.835} -> T^{-1} | Helped |
| Use state-wise average policies rather than the last iterate | Algorithm framework for two-player zero-sum Markov games · Two-player zero-sum Markov games, full-information | NEGap | Helped |
Closest papers by meaning
- Faster Last-iterate Convergence of Policy Optimization in Zero-Sum Markov Games (ICLR 2023 · similarity 0.94)
- Optimistic Policy Gradient in Multi-Player Markov Games with a Single Controller: Convergence beyond the Minty Property (AAAI 2024 · similarity 0.93)
- Improving Sample Efficiency of Model-Free Algorithms for Zero-Sum Markov Games (ICML 2024 · similarity 0.91)
- Solving Zero-Sum Convex Markov Games (ICML 2025 · similarity 0.91)
- Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Games (ICML 2025 · similarity 0.91)
- Learning Equilibria in Adversarial Team Markov Games: A Nonconvex-Hidden-Concave Min-Max Optimization Problem (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.