Research map / Multi-agent systems and games / Cooperative multi-agent RL
AgentMixer: Multi-Agent Correlated Policy Factorization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AgentMixer, which non-linearly combines decentralized partially observable policies into a correlated joint fully observable policy via a Policy Modifier and keeps them consistent with Individual-Global-Consistency, converging to an epsilon-approximate correlated equilibrium.
Problem
Cooperative MARL under CTDE: independent decentralized policies cannot express correlated coordination, and distilling a centralized fully observable expert into partially observable agents causes an asymmetric learning failure.
Approach
A Policy Modifier built from MLP agent-mixing and channel-mixing layers produces a correlated joint policy from individual policies and state, and the Individual-Global-Consistency constraint forces the joint policy's modes to match the decentralized policies' modes so that joint training yields decentralized policies approximating a correlated equilibrium.
Key results
- On SMAC-v2 5v5 maps, AgentMixer reaches 0.68 (protoss), 0.71 (terran), and 0.66 (zerg) median win rate, outperforming the listed baselines in those scenarios.
- On SMAC-v2 10v10 maps, AgentMixer reaches 0.47, 0.50, and 0.52 win rates, with MAT or MAT-Dec matching or exceeding it on some maps.
- On the Ant-v2 partial-observation ablation, AIL shows a large train-test gap while AgentMixer obtains superior performance; other baselines fail to learn effective policies.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AgentMixer instead of MAT | SMAC-v2 protoss 5v5 · SMAC-v2 protoss 5v5 | winning rate: 0.55_{0.11} -> 0.68_{0.04} | Helped |
| use AgentMixer instead of MAT | SMAC-v2 terran 5v5 · SMAC-v2 terran 5v5 | winning rate: 0.63_{0.11} -> 0.71_{0.10} | Helped |
| use AgentMixer instead of MAT-Dec | SMAC-v2 zerg 5v5 · SMAC-v2 zerg 5v5 | winning rate: 0.51_{0.10} -> 0.66_{0.03} | Helped |
| use AgentMixer instead of MAT-Dec | SMAC-v2 protoss 10v10 · SMAC-v2 protoss 10v10 | winning rate: 0.52_{0.02} -> 0.47_{0.02} | Hurt |
| use AgentMixer instead of MAT | SMAC-v2 terran 10v10 · SMAC-v2 terran 10v10 | winning rate: 0.46_{0.14} -> 0.50_{0.11} | Helped |
| use AgentMixer instead of MAT | SMAC-v2 zerg 10v10 · SMAC-v2 zerg 10v10 | winning rate: 0.52_{0.09} -> 0.52_{0.06} | No clear effect |
| replace the MLP Policy Modifier with hypernetworks (ablation) | AgentMixer · SMAC-v2 protoss 5v5 | winning rate: 0.68_{0.04} -> 0.60_{0.13} | Hurt |
| replace the MLP Policy Modifier with attention (ablation) | AgentMixer · SMAC-v2 protoss 5v5 | winning rate: 0.68_{0.04} -> 0.59_{0.14} | Hurt |
| replace the MLP Policy Modifier with hypernetworks (ablation) | AgentMixer · SMAC-v2 terran 5v5 | winning rate: 0.71_{0.10} -> 0.53_{0.13} | Hurt |
| replace the MLP Policy Modifier with attention (ablation) | AgentMixer · SMAC-v2 terran 5v5 | winning rate: 0.71_{0.10} -> 0.60_{0.02} | Hurt |
| replace the MLP Policy Modifier with hypernetworks (ablation) | AgentMixer · SMAC-v2 zerg 5v5 | winning rate: 0.66_{0.03} -> 0.52_{0.08} | Hurt |
| replace the MLP Policy Modifier with attention (ablation) | AgentMixer · SMAC-v2 zerg 5v5 | winning rate: 0.66_{0.03} -> 0.49_{0.10} | Hurt |
Surprising findings
- Partially observable policies achieve similar performance to fully observable MAPPO_FULL on SMACv2, suggesting global information is not important there.
- AgentMixer's run time is substantially higher than baselines (e.g., 60737 s vs 24070 s for MAPPO on 10v10 zerg).
Limitations
- Approximate convergence to epsilon-CE relies on assumptions (sufficient expressiveness, exact updates) and the proof is tied to a single-agent RL reduction with a sub-optimal stationary point.
- Higher computational cost than baselines in run-time comparison.
- Only MAPPO-based implementation is used for the main method; integration with other algorithms (SAC, QMIX) is only partially explored.
Experimental setup
Datasets: MA-MuJoCo SMAC-v2 Climbing Matrix Game Predator-Prey
Benchmarks: MA-MuJoCo SMAC-v2 Climbing Matrix Game Predator-Prey
Metrics: win rate episode return statistical significance (t-test p-value)
Baselines: MAPPO HAPPO MAT MAT-Dec MAVEN ARMAPPO AIL
Builds on: PPO MAPPO MAT QMIX MAVEN
Closest papers by meaning
- More Centralized Training, Still Decentralized Execution: Multi-Agent Conditional Policy Factorization (ICLR 2023 · similarity 0.89)
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (ICML 2018 · similarity 0.89)
- FACMAC: Factored Multi-Agent Centralised Policy Gradients (NeurIPS 2021 · similarity 0.88)
- Submodular Multi-Agent Reinforcement Learning for Effective Online Distributed Task Allocation (NeurIPS 2026 · similarity 0.88)
- Backpropagation Through Agents (AAAI 2024 · similarity 0.87)
- Intrinsic Action Tendency Consistency for Cooperative Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.