Research map / Multi-agent systems and games / Cooperative multi-agent RL
Robust and Diverse Multi-Agent Learning via Rational Policy Gradient
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Rationality-preserving Policy Optimization (RPO) and Rational Policy Gradient (RPG), using manipulator agents with opponent shaping to extend adversarial optimization algorithms (AP, AT, PAIRED, AD) to cooperative and general-sum games without self-sabotage.
Problem
Adversarial optimization works in zero-sum games but in cooperative/general-sum settings it leads to self-sabotage, where agents irrationally minimize their own reward, blocking task completion and learning.
Approach
Require each adversarially-optimized policy to be a best response to some co-policy (RPO), enforced by training base agents against manipulator agents that use opponent shaping (higher-order gradients via Loaded DiCE) to steer base agents toward adversarial objectives while base agents only maximize their own reward.
Key results
- In cramped room Overcooked, AP-RPG finds an adversarial policy achieving reward 4.6 against a self-play victim with reward 240, while AP collapses to self-sabotage.
- AD-RPG maintains high cross-play reward in Overcooked where CoMeDi and AD agents achieve low cross-play through sabotage (standing in front of plate dispenser).
- In modified STORM, CoMeDi learns to sabotage with large negative cross-play reward while AD-RPG avoids the sabotage state in both self-play and cross-play.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AP-RPG instead of self-play to find a vulnerability in the fixed policy | Cramped Room Overcooked self-play victim · Cramped Room Overcooked | reward: 240 -> 4.6 | Hurt |
| Use AP-RPG instead of AP | Victims in unobserved STORM · Unobserved STORM | reward against adversarial attack: zero | Mixed |
| Use PAIRED-RPG or AT-RPG to train the victim | Victim policies · Unobserved STORM | reward against adversarial attacks | Helped |
| Use RPG to preserve rationality in adversarial diversity training | AD-RPG · STORM and Overcooked | cross-play reward | Helped |
| Use AD-RPG instead of SP or AD | Policies trained for partner robustness · Overcooked | intra-population cross-play reward | Helped |
| Use SP with high entropy coefficient instead of low entropy coefficient | Self-play policies · 3-color and 4-color Hanabi | intra-population cross-play reward | Helped |
| Add RPG manipulators to AD | AD training · Training runtime | relative wall-clock time: three times slower | Hurt |
| Use a larger lookahead in RPG | AT-RPG · Matrix game | learning stability | Helped |
| Use lookahead greater than 8 | AT-RPG · Rock-paper-scissors matrix game | learning stability: greater than 8 | Helped |
Surprising findings
- Naive adversarial diversity and CoMeDi cross-play low scores in Overcooked are driven by sabotage rather than genuine diversity; the cramped room layout contains little genuine diversity.
- Standard self-play with high entropy often converged to similar strategies, yielding higher intra-population reward than low entropy in Hanabi, which the authors found somewhat surprising.
Limitations
- Higher-order gradient estimation through learning updates has high variance and computational overhead, requiring large batch sizes.
- No formal convergence guarantee to truly rational strategies.
- Experiments limited to small environments with small MLP policies (64x64 or 512x512).
Experimental setup
Datasets: STORM Overcooked Hanabi matrix games
Benchmarks: STORM unobserved STORM Overcooked Hanabi
Metrics: self-play reward cross-play reward intra-population cross-play reward adversarial attack reward
Baselines: self-play adversarial policy adversarial training PAIRED adversarial diversity CoMeDi
Builds on: opponent shaping (LOLA) Loaded DiCE adversarial policy PAIRED adversarial diversity
Closest papers by meaning
- Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula (ICLR 2024 · similarity 0.90)
- Rethinking Adversarial Policies: A Generalized Attack Formulation and Provable Defense in RL (ICLR 2024 · similarity 0.89)
- Robust Multi-Agent Reinforcement Learning with Stochastic Adversary (ICML 2025 · similarity 0.89)
- Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game (ICLR 2024 · similarity 0.89)
- Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation (ICML 2025 · similarity 0.88)
- DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.