Research map / Multi-agent systems and games / Cooperative multi-agent RL
N-agent Ad Hoc Teamwork
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces N-agent ad hoc teamwork (NAHT), where a learner controls a varying subset of agents among uncontrolled teammates, and proposes POAM, a policy-gradient MARL method with an encoder-decoder teammate model and IPPO-style training that also uses uncontrolled agents' data for the critic.
Problem
Cooperative learning where a learning algorithm controls a varying number N of agents in open teams that also include uncontrolled teammates with unknown behaviors; existing CMARL assumes full control and AHT assumes single-agent control.
Approach
Generalizes AHT and CMARL by letting a shared policy control a random subset of agents; POAM learns an encoder-decoder agent-modeling network producing team embeddings that condition an IPPO-based actor-critic, and trains the critic on data from both controlled and uncontrolled agents.
Key results
- POAM outperforms baselines in asymptotic return on three of five tasks (mpe-pp, 5v6, 3s5z) and improves sample efficiency over baselines, with IPPO-NAHT converging to similar returns on 8v9 and 10v11.
- On the three-agent bit matrix game, POAM achieves 48.858 ± 1.092 return in the N=2 NAHT setting versus 22.5 ± 8.250 for POAM-AHT, matching the theoretical optimum of 50.
- POAM shows higher returns than IPPO-NAHT when paired with out-of-distribution teammates on three of five tasks (mpe-pp, 5v6, 3s5z).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the policy for varying numbers of controlled agents (NAHT) instead of always using one controlled agent (AHT) | POAM · Three-agent bit matrix game, N=2 (NAHT) | return: 22.5 $\pm$ 8.250 -> 48.858 $\pm$ 1.092 | Helped |
| Train the policy for varying numbers of controlled agents (NAHT) instead of always using one controlled agent (AHT) | POAM · Three-agent bit matrix game, N=1 (AHT) | return: 33.441 $\pm$ 0.511 -> 33.483 $\pm$ 0.485 | No clear effect |
| Remove the agent modeling module (ablation) | POAM · mpe-pp, 5v6, 8v9, 10v11, 3s5z | return | Mixed |
| Train the value network using uncontrolled agents' data, rather than on-policy data only (ablation) | POAM and IPPO-NAHT · mpe-pp, 5v6 | return | Helped |
Surprising findings
- Naive MARL algorithms perform much better in self-play than in cross-play with teams trained by other algorithms or seeds, indicating non-compatible coordination conventions.
- A single-agent AHT policy copied across agents (POAM-AHT) can be substantially suboptimal in NAHT settings, even though it is optimal in the N=1 case.
Limitations
- Only homogeneous, non-communicating controlled agents with uniform team sampling are studied.
- POAM's actor is purely on-policy and cannot exploit uncontrolled agents' data; full parameter sharing may fail with heterogeneous roles.
- Results are reported over five trials on four SMAC maps and one custom MPE task.
Experimental setup
Datasets: MPE predator-prey (mpe-pp) SMAC 5v6 SMAC 8v9 SMAC 10v11 SMAC 3s5z
Benchmarks: MPE predator-prey (mpe-pp) SMAC 5v6 SMAC 8v9 SMAC 10v11 SMAC 3s5z
Metrics: test return M-N score cross-play score 95% confidence interval
Baselines: IQL VDN QMIX MAPPO IPPO-NAHT POAM-AHT (LIAM)
Builds on: IPPO PPO LIAM ePyMARL
Closest papers by meaning
- TEAMSTER: Model-Based Reinforcement Learning for Ad Hoc Teamwork (Abstract Reprint) (AAAI 2024 · similarity 0.92)
- Open Ad Hoc Teamwork with Cooperative Game Theory (ICML 2024 · similarity 0.90)
- ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork (NeurIPS 2026 · similarity 0.89)
- Minimum Coverage Sets for Training Robust Ad Hoc Teamwork Agents (AAAI 2024 · similarity 0.88)
- Ad Hoc Teamwork via Offline Goal-Based Decision Transformers (ICML 2025 · similarity 0.88)
- POGEMA: A Benchmark Platform for Cooperative Multi-Agent Pathfinding (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.