Research map / Multi-agent systems and games / Cooperative multi-agent RL
TAPE: Leveraging Agent Topology for Cooperative Multi-Agent Policy Gradient
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAPE, which uses a random agent topology (Erdős–Rényi graph) to define coalition-based utilities for policy gradients, balancing cooperation against the centralized-decentralized mismatch in both stochastic and deterministic MAPG.
Problem
Centralized critics in multi-agent policy gradient suffer from the centralized-decentralized mismatch (sub-optimal actions of some agents corrupt others' learning), while individual critics limit cooperation.
Approach
An agent topology (adjacency matrix from random graphs, mainly Erdős–Rényi) specifies which agents' utilities each agent includes in its policy gradient, giving coalition utilities that sit between fully centralized and fully individual objectives; the topology density p trades off cooperation against CDM.
Key results
- On SMAC 6 maps, deterministic TAPE outperforms all compared methods in performance and convergence speed, including PAC (its base method), per the paper's overall results.
- Stochastic TAPE achieves best performance on LBF 8x8-2p-3f, and deterministic TAPE achieves best performance on LBF 15x15-4p-5f where only PAC and TAPE learn to collect food.
- Theorem 2 shows Var of TAPE policy update exceeds DOP's by a term proportional to p^2 in the Erdős–Rényi model.
Surprising findings
- Fully-connected topology (centralized critic) shows very large variance on MMM2, since one sub-optimal action can spread to all agents.
- Graph-search results show the optimal edge density depends on CDM severity: more edges preferred at small p, fewer at large p (p=0.3).
Limitations
- Topologies are generated from existing random graph models rather than learned; adaptive topology learning is left to future work.
- Only p is tuned; the policy improvement theorem is proven for tabular policies, with only informal extension to function approximators.
Experimental setup
Datasets: Level-Based Foraging StarCraft Multi-Agent Challenge three one-step matrix games
Benchmarks: SMAC Level-Based Foraging one-step matrix games
Metrics: test win rate average test return normalized performance
Baselines: DOP COMA PAC QMIX QPLEX FACMAC InfoPG
Builds on: DOP COMA PAC QMIX FACMAC
Closest papers by meaning
- Optimistic Multi-Agent Policy Gradient (ICML 2024 · similarity 0.88)
- Graph Diffusion for Robust Multi-Agent Coordination (ICML 2025 · similarity 0.88)
- Learn to Follow: Decentralized Lifelong Multi-Agent Pathfinding via Planning and Learning (AAAI 2024 · similarity 0.87)
- Multi-agent cooperation through learning-aware policy gradients (ICLR 2025 · similarity 0.86)
- POGEMA: A Benchmark Platform for Cooperative Multi-Agent Pathfinding (ICLR 2025 · similarity 0.86)
- Bridging Training and Execution via Dynamic Directed Graph-Based Communication in Cooperative Multi-Agent Systems (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.