Research map / Multi-agent systems and games / Cooperative multi-agent RL
Multi-agent cooperation through learning-aware policy gradients
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COALA-PG, an unbiased, higher-derivative-free policy gradient for learning-aware agents that shape co-players' minibatched learning, and uses Hawk sequence policies to get cooperation in the IPD and a CleanUp-lite social dilemma.
Problem
Independent self-interested learners often fail to cooperate in general-sum social dilemmas; existing learning-aware methods (LOLA, M-FOS) either need higher-order derivatives or yield biased gradients when co-players learn from minibatches.
Approach
Reformulate co-player shaping as a batched POMDP in which B parallel trajectories share co-player minibatch updates, and derive a policy gradient that sums (rather than averages) per-action gradients, rescaling current-episode returns by 1/B, enabling a model-free, unbiased learning-aware update applied with long-context sequence policies.
Key results
- In the IPD with mixed naive/learning-aware pools, COALA-PG agents achieve robust cooperation, whereas pure learning-aware groups stay in mutual defection (32 seeds, interquartile range reported).
- M-FOS and the batch-unaware naive policy gradient fail to escape defection with Hawk policies in the IPD, while COALA-PG succeeds.
- On CleanUp-lite, COALA-PG agents reach significantly higher returns than model-free baselines, shaping naive opponents to zap less and clean more.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COALA-PG for learning-aware policy updates | Hawk sequence policies · IPD | cooperation | Helped |
| Use LOLA-DICE instead of COALA-PG | Hawk sequence policies · IPD | cooperation | Helped |
| Use M-FOS instead of COALA-PG | Hawk sequence policies · IPD | cooperation | Hurt |
| Ignore co-player batching in the policy gradient (ablation) | COALA-PG · IPD | cooperation | Hurt |
| Use COALA-PG instead of previous model-free baselines | Hawk sequence policies · CleanUp-lite | return | Helped |
| Include naive agents in a mixed group with learning-aware agents | IPD learning-aware agents · IPD | cooperation | Helped |
| Train in a pure group of learning-aware agents rather than a mixed group | IPD learning-aware agents · IPD | cooperation | Hurt |
| Increase LOLA-DICE look-ahead steps (ablation) | LOLA-DICE agents · IPD | cooperation | Hurt |
| Add the partial gradient against the current co-player policy (ablation) | LOLA with many look-ahead steps · IPD | performance | Helped |
| Train against COALA-PG agents rather than other baselines | Naive agents · CleanUp-lite | average reward | Helped |
Surprising findings
- Mutual unconditional defection is not a Nash equilibrium in a mixed naive/learning-aware group, so agents initialized at defection learn to cooperate.
- A pure group of learning-aware agents cannot escape mutual defection, while adding naive learners enables cooperation.
Limitations
- Shaping guarantees rely on naive co-players and fixed learning rules; asymmetric setup not always realistic.
- Experiments limited to IPD and a simplified two-player CleanUp-lite environment.
- Naive agents in CleanUp-lite are initialized from already-capable dynamic copies to avoid prohibitively slow learning from scratch.
Experimental setup
Datasets: iterated prisoner's dilemma (simulated) CleanUp-lite (adapted from Melting Pot)
Benchmarks: iterated prisoner's dilemma CleanUp-lite
Metrics: episode return cooperation rate zapping rate pollution level apple harvest
Baselines: LOLA-DICE M-FOS naive policy gradient (batch-unaware COALA)
Builds on: LOLA LOLA-DICE M-FOS Shaper Hawk
Closest papers by meaning
- Self-Play $Q$-Learners Can Provably Collude in the Iterated Prisoner's Dilemma (ICML 2025 · similarity 0.89)
- Cautiously-Optimistic Knowledge Sharing for Cooperative Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.89)
- The Horizon Threshold in Cooperative Multi-Agent Reward-Free Exploration (NeurIPS 2026 · similarity 0.89)
- Fair Cooperation in Mixed-Motive Games via Conflict-Aware Gradient Adjustment (NeurIPS 2025 · similarity 0.89)
- Reciprocal Reward Influence Encourages Cooperation From Self-Interested Agents (NeurIPS 2024 · similarity 0.88)
- Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.