Research map / Multi-agent systems and games / Cooperative multi-agent RL
AIR: Unifying Individual and Collective Exploration in Cooperative Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AIR, an adversarial identity-recognition method that unifies individual and collective exploration for value-based cooperative MARL, with an adaptive temperature controlling exploration mode and degree.
Problem
Exploration for value-based agents in cooperative MARL is hard because there is no explicit policy; existing methods either explore individually (uncertainty/curiosity) or collectively (behavioral diversity), and combining them with extra modules makes training heavy and inefficient.
Approach
A centralized identity classifier recognizes which agent produced a trajectory-action pair, and a decentralized action selector adds a temperature-scaled negative log-posterior term to Q-values so that a positive temperature drives individual exploration and a negative one drives collective (diversity) exploration; the temperature is tuned adaptively via dual gradient descent under a minimum-entro
Key results
- AIR achieves the best performance in 5 of 6 hard/super-hard SMAC scenarios, with the highest win rates and earliest rise-ups; RODE is slightly better on corridor.
- On three GRF academy scenarios, AIR performs well on both harder scenarios (academy_corner and academy_run_pass_and_shoot_with_keeper) where most baselines succeed on only one.
- AIR uses an extremely lightweight model relative to QMIX (model size reported relative to QMIX at 100%), and in SMACv2 it exceeds the highest win rate records reported in the original SMACv2 paper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AIR exploration with QMIX | QMIX · SMAC Hard and Super Hard scenarios | test win rate | Helped |
| use AIR exploration with QMIX | QMIX · SMAC corridor | test win rate | Hurt |
| use AIR exploration with QMIX | QMIX · Google Research Football Academy scenarios | performance | Helped |
| use AIR exploration with QMIX | QMIX · Academy_pass_and_shoot_with_keeper | performance | No clear effect |
| use AIR exploration with QMIX | QMIX · SMACv2 scenarios | win rate | Helped |
| add individual exploration only (constrain temperature to be positive) (ablation) | AIR · SMAC MMM2 and 6h_vs_8z | performance | Hurt |
| add collective exploration only (constrain temperature to be negative) (ablation) | AIR · SMAC MMM2 and 6h_vs_8z | performance | Hurt |
| extend the epsilon anneal period from 100K to 500K (ablation) | QMIX with epsilon-greedy exploration · SMAC 6h_vs_8z and corridor | performance | Mixed |
Surprising findings
- RODE outperforms AIR on SMAC corridor but degrades substantially on GRF, while AIR stays superior in both.
- Fixed epsilon-greedy QMIX with more exploration helps on 6h_vs_8z but completely fails on corridor, showing that a single exploration intensity is not robust across tasks.
Limitations
- Theoretical guarantees rely on approximations (classifier as posterior, truncated dual updates) that the authors acknowledge are not strictly satisfied by neural networks.
- Experiments use SMAC version 4.6.2 and a limited set of scenarios; AIR uses a parallel runner that reduces network update frequency, which the authors note may affect comparisons with baselines on the episode runner.
Experimental setup
Datasets: SMAC SMACv2 Google Research Football
Benchmarks: SMAC SMACv2 Google Research Football
Metrics: test win rate score/goal performance relative model size
Baselines: QMIX QPLEX MAVEN RODE LDSA EMC QMIX with epsilon-greedy annealing variants
Builds on: QMIX soft Q-learning MAVEN CDS dual gradient descent
Closest papers by meaning
- Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration (ICML 2025 · similarity 0.88)
- The Horizon Threshold in Cooperative Multi-Agent Reward-Free Exploration (NeurIPS 2026 · similarity 0.87)
- Learning Joint Behaviors with Large Variations (AAAI 2025 · similarity 0.86)
- Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learning (ICML 2024 · similarity 0.86)
- Settling Decentralized Multi-Agent Coordinated Exploration by Novelty Sharing (AAAI 2024 · similarity 0.86)
- Robust and Diverse Multi-Agent Learning via Rational Policy Gradient (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.