Research map / Multi-agent systems and games / Cooperative multi-agent RL
Settling the Variance of Multi-Agent Policy Gradients
Jakub Grudzien Kuba, Muning Wen, Linghui Meng, Shangding Gu, Haifeng Zhang, David Henry Mguni, Jun Wang, Yaodong Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add optimal baseline (OB) to multi-agent PPO (ablation) | MAPPO · SMAC 3s vs. 5z | std of gradient norm (x0.01): 6.67 -> 2.20 | Improved |
| add OB to MAPPO (ablation) | MAPPO · SMAC 27m vs. 30m | std of gradient norm (x0.01): 19.62 -> 11.55 | Improved |
| add OB to MAPPO (ablation) | MAPPO · SMAC 5m vs. 6m | std of gradient norm (x0.01): 2.17 -> 1.50 | Improved |
| add OB to MAPPO (ablation) | MAPPO · SMAC 6h vs. 8z | std of gradient norm (x0.01): 2.54 -> 2.33 | Improved |
| add OB to MAPPO (ablation) | MAPPO · MaMuJoCo 2-Agent Walker | std of gradient norm (x0.01): 405.66 -> 372.03 | Improved |
| add OB to MAPPO (ablation) | MAPPO · MaMuJoCo 6-Agent HalfCheetah | std of gradient norm (x0.01): 33.65 -> 30.64 | Improved |
| add OB to MAPPO (ablation) | MAPPO · SMAC hard map | win rate: 0 -> 90% | Improved |
| add OB to MAPPO (ablation) | MAPPO · Multi-Agent MuJoCo | return | Mixed |
| add OB to COMA (ablation) | COMA · SMAC easy maps | reward / stability | Improved |
| use OB instead of COMA counterfactual baseline | toy example · toy example | gradient estimator variance: 1015 -> 673 | Improved |
| use COMA counterfactual baseline instead of none (vanilla MAPG) | toy example · toy example | gradient estimator variance: 1321 -> 1015 | Improved |
Closest papers by meaning
- On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Method (NeurIPS 2021 · similarity 0.89)
- Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines (ICLR 2018 · similarity 0.89)
- Marginal Policy Gradients: A Unified Family of Estimators for Bounded Action Spaces with Applications (ICLR 2019 · similarity 0.89)
- Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning (ICML 2022 · similarity 0.88)
- Optimistic Multi-Agent Policy Gradient (ICML 2024 · similarity 0.88)
- Convergence Analysis of Policy Gradient Methods with Dynamic Stochasticity (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
