Research map / Multi-agent systems and games / Cooperative multi-agent RL
Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ICES, which uses Bayesian surprise from a two-encoder CVAE during centralized training to give each agent an individual exploration bonus, and decouples a separate exploration policy from the QMIX/QPLEX exploitation policy.
Problem
Exploration in cooperative multi-agent RL under sparse rewards is hard, and global intrinsic rewards complicate credit assignment and add non-stationarity to the training objective.
Approach
Replace global intrinsic rewards with individual contribution scores, measured as the KL divergence between latent state-transition priors with and without agent i's action (Bayesian surprise), computed by a CVAE with two prior encoders and a shared decoder. Exploration policies are trained separately with REINFORCE plus a baseline and may use global state, and they are used only during training w
Key results
- On GRF academy_3_vs_1_with_keeper, ICES reaches over 60% winning rate while the global-contribution variant (ICES w/ global-con) reaches about 40%.
- On SMAC 2s_vs_1sc, ICES starts observing winning episodes within 2M timesteps while baselines have not.
- On SMAC, removing global observation from the exploration policy (ICES w/o s) drops the final win rate to about 20% from 60% for ICES in the 5m_vs_6m task (as stated in the paper's ablation text).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace individual contributions with a global contribution scaffold (ablation) | ICES · GRF 3_vs_1_with_keeper | winning rate: over 60% -> 40% | Hurt |
| sum individual scaffolds into a global intrinsic reward (ablation) | ICES · GRF | exploration effectiveness | Hurt |
| exclude the global observation from the exploration policy inputs (ablation) | ICES · SMAC | final winning rate: 60% -> 20% | Hurt |
| merge exploration and exploitation policies into a single network (ablation) | ICES · SMAC | exploration effectiveness | Hurt |
| remove entropy regularization by setting β = 0 (ablation) | ICES · representative GRF and SMAC tasks | final performance or convergence speed | Hurt |
| replace latent-space KL divergence with Euclidean distance in the original state space (ablation) | ICES · representative GRF and SMAC tasks | final performance or convergence speed | Hurt |
| use two independent CVAEs instead of two encoders with a shared decoder (ablation) | ICES · representative GRF and SMAC tasks | final performance or convergence speed | Hurt |
| use individual-contribution scaffolds for exploration | QMIX-based MARL · GRF and SMAC sparse-reward tasks | performance | Helped |
| use individual-contribution scaffolds for exploration | QPLEX-based MARL · GRF | performance | Helped |
| use ICES exploration scaffolds | QMIX · KAZ | performance | Helped |
Surprising findings
- Summing individual scaffolds into a global intrinsic reward (ICES w/ int+ext) further degrades performance, attributed to added credit-assignment complexity.
Limitations
- Requires additional policy networks, increasing training complexity relative to QMIX or QPLEX.
- Only one-step latent state transitions are considered, which may be insufficient for complex scenarios.
Experimental setup
Datasets: Google Research Football (academy_3_vs_1_with_keeper, academy_corner, academy_counterattack_hard) StarCraft Multi-agent Challenge (3m, 8m, 2s3z, 2s_vs_1sc, 5m_vs_6m) knights_archers_zombies (PettingZoo)
Benchmarks: Google Research Football StarCraft Multi-agent Challenge knights_archers_zombies
Metrics: average score / win rate over training timesteps winning rate convergence speed
Baselines: ADER MASER EMC CDS MAVEN QMIX QPLEX
Builds on: QMIX QPLEX PyMARL REINFORCE
Closest papers by meaning
- Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibration (NeurIPS 2025 · similarity 0.89)
- Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.88)
- Settling Decentralized Multi-Agent Coordinated Exploration by Novelty Sharing (AAAI 2024 · similarity 0.88)
- Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration (NeurIPS 2026 · similarity 0.88)
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Action-Dependent Optimality-Preserving Reward Shaping (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.