Research map / Multi-agent systems and games / Cooperative multi-agent RL
Optimistic Multi-Agent Policy Gradient
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OptiMAPPO, which clips negative advantages in MAPPO to inject optimism into multi-agent policy gradients, mitigating relative overgeneralization in cooperative MARL.
Problem
Relative overgeneralization in cooperative multi-agent learning, where agents converge to suboptimal joint policies; this has not been addressed in multi-agent policy gradient methods.
Approach
Clip the advantage estimates at zero (generalized to a Leaky ReLU with slope eta) when updating MAPPO policies, so that actions with negative advantage are not penalized, giving on-policy optimism without the overestimation of optimistic Q-learning; a fixed-point analysis shows optimality is retained.
Key results
- On repeated matrix games, OptiMAPPO reaches the global optimum (275 on Climbing, 250 on all Penalty variants) while MAPPO, HAPPO and HATRPO get stuck at 50-175.
- On MA-MuJoCo, OptiMAPPO beats FACMAC by a wide margin on nine tasks (e.g., Ant 8x1 6393 vs 1953; Walker 6x1 4957 vs 950).
- The abstract claims OptiMAPPO outperforms strong baselines on 13 of 19 tested tasks and matches performance on the rest.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| clip negative advantages in MAPPO updates | MAPPO on Climbing · Climbing | average return: 175 -> 275 | Helped |
| clip negative advantages in MAPPO updates | MAPPO on Penalty games · Penalty k=-25, -50, -75, -100 | average return: 50 -> 250 | Helped |
| clip negative advantages in MAPPO updates | MAPPO on Penalty game · Penalty k=0 | average return: 250 -> 250 | No clear effect |
| clip negative advantages in MAPPO updates | MAPPO on MA-MuJoCo · MA-MuJoCo tasks in FACMAC comparison | average episode return: 307.58 (78.28) -> 6103.97 (180.62) | Helped |
| clip negative advantages in MAPPO updates | MAPPO on Penalized Push-Box · Penalized Push-Box | episode return: 0 -> 1.6 | Helped |
| vary Leaky ReLU negative slope η from 0 toward 1 (less optimism) (ablation) | OptiMAPPO on MA-MuJoCo · HalfCheetah 6x1 | average episode return | Hurt |
| vary Leaky ReLU negative slope η from 0 toward 1 (less optimism) (ablation) | OptiMAPPO on Overcooked · Random 3 | average episode return | No clear effect |
| increase optimism in hysteretic DQN by lowering α (ablation) | Hysteretic DQN on Overcooked · Overcooked tasks | episode return | Hurt |
Surprising findings
- Q-learning-based optimism (hysteretic DQN) amplifies overestimation and degrades performance, whereas on-policy advantage clipping avoids this.
- General exploration methods (MAVEN, NA-MAPPO) fail to solve the relative overgeneralization problem on matrix games.
Limitations
- Optimism can converge to suboptimal solutions when misleading stochastic rewards exist; the study is limited to deterministic-style environments.
- Clipping at eta=0 discards information from negative-advantage data, causing slower early learning on MA-MuJoCo.
Experimental setup
Datasets: climbing matrix game penalty matrix game Multi-agent MuJoCo Overcooked Penalized Push-Box
Benchmarks: Climbing matrix game Penalty matrix game Multi-agent MuJoCo Overcooked Penalized Push-Box
Metrics: average episode return maximum episode return
Baselines: MAPPO HAPPO HATRPO FACMAC Hysteretic DQN MAVEN NA-MAPPO CMAE
Builds on: MAPPO PPO hysteretic Q-learning FACMAC HAPPO
Closest papers by meaning
- HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.88)
- Measuring Mutual Policy Divergence for Multi-Agent Sequential Exploration (NeurIPS 2024 · similarity 0.88)
- TAPE: Leveraging Agent Topology for Cooperative Multi-Agent Policy Gradient (AAAI 2024 · similarity 0.88)
- Robust and Diverse Multi-Agent Learning via Rational Policy Gradient (NeurIPS 2025 · similarity 0.88)
- Optimistic Value Instructors for Cooperative Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.88)
- Multi-agent cooperation through learning-aware policy gradients (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.