Research map / Multi-agent systems and games / Cooperative multi-agent RL
Attention-Guided Contrastive Role Representations for Multi-agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACORM, which learns contrastive role representations via an InfoNCE objective over K-means clusters of agent embeddings and feeds them to QMIX's mixing network through multi-head attention conditioned on the global state, evaluated on SMAC and GRF.
Problem
Cooperative MARL with parameter sharing tends to produce homogeneous agents, and existing role methods either need domain knowledge, use fixed roles, or relax CTDE; the paper targets dynamic role emergence that promotes behavior heterogeneity and coordination.
Approach
Learn role representations by maximizing a mutual-information lower bound with an InfoNCE loss, where negatives come from periodically K-means clustering agent GRU embeddings, and use multi-head attention with the global state as query over role representations inside the QMIX mixing network.
Key results
- ACORM achieves best performance on all super-hard SMAC maps (MMM2, 3s5z_vs_3s6z, corridor) and most other maps, with faster early win-rate growth than baselines (five seeds).
- On three GRF academy scenarios, ACORM improves over QMIX and CDS in test win rate within 2M steps (only QMIX and CDS compared).
- Ablations show removing contrastive learning or attention each reduces performance but both still outperform QMIX on super-hard maps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ACORM contrastive role learning and attention-guided coordination | QMIX · SMAC super-hard maps (MMM2, 3s5z_vs_3s6z, corridor) | test win rate | Helped |
| Add ACORM contrastive role learning and attention-guided coordination | QMIX · SMAC representative maps | test win rate | Helped |
| Add ACORM role learning and attention-guided coordination | QMIX · Google Research Football offensive scenarios | test win rate | Helped |
| Add ACORM role learning and attention-guided coordination | MAPPO · SMAC representative maps | test win rate | Helped |
| Add contrastive role learning (ablation) | ACORM (QMIX) · SMAC super-hard maps | test win rate | Helped |
| Add attention-guided role coordination (ablation) | ACORM (QMIX) · SMAC super-hard maps | test win rate | Helped |
| Remove attention and state encoding rather than removing attention alone (ablation) | ACORM (QMIX) · SMAC representative maps | test win rate | No clear effect |
| Add contrastive learning (ablation) | ACORM (QMIX) · SMAC representative maps | learning-curve variance | Helped |
| Vary the number of role clusters across K=2, 3, 4, and 5 (ablation) | ACORM (QMIX) · SMAC maps generally | test win rate | No clear effect |
| Set the number of role clusters to K=5 (ablation) | ACORM (QMIX) · SMAC 5m_vs_6m | test win rate | Hurt |
Surprising findings
- ACORM without the attention module but with the GRU state encoding performs very similarly to the vanilla variant, attributing gains to attention rather than state encoding.
Limitations
- Requires an explicit clustering component with a pre-defined number of clusters K.
- Does not exploit historical exploratory trajectories for extracting roles.
- Only online settings evaluated; offline extension left to future work.
Experimental setup
Datasets: SMAC Google Research Football
Benchmarks: SMAC Google Research Football
Metrics: test win rate
Baselines: QMIX RODE EOI MACC CDS CIA GoMARL ROMA COPA
Builds on: QMIX MAPPO MoCo InfoNCE CIA
Closest papers by meaning
- GRDC: A Unified Graph-Driven Framework for Role Discovery and Communication in Multi-Agent Reinforcement Learning (AAAI 2026 · similarity 0.89)
- ROMA: Multi-Agent Reinforcement Learning with Emergent Roles (ICML 2020 · similarity 0.89)
- R3DM: Enabling Role Discovery and Diversity Through Dynamics Models in Multi-agent Reinforcement Learning (ICML 2025 · similarity 0.89)
- Effective and Stable Role-Based Multi-Agent Collaboration by Structural Information Principles (AAAI 2023 · similarity 0.88)
- RODE: Learning Roles to Decompose Multi-Agent Tasks (ICLR 2021 · similarity 0.88)
- Acting Beyond Learning: Imagination-Assisted Decision-Making in the Visual-based Multi-Agent Cooperative Scenarios (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.