Research map / Multi-agent systems and games / Cooperative multi-agent RL
Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CERMIC, a plug-and-play module that calibrates intrinsic curiosity in communication-free MARL using an inferred multi-agent intention graph, an information-bottleneck objective with distributionally robust chance constraints, and a Bayesian-surprise-style intrinsic reward.
Problem
Exploration in sparse-reward, partially observable, communication-free multi-agent RL, where curiosity signals confuse environmental stochasticity with meaningful novelty and treat all surprises uniformly.
Approach
Calibrate the mean of a curiosity-based novelty measure with an inferred multi-agent context (graph-based intention model) via distributionally robust chance constraints derived from the Information Bottleneck, with the calibration weight set by InfoNCE-estimated mutual information between context and intention; the intrinsic reward is a KL-based Bayesian surprise linked to LSVI-UCB bonuses.
Key results
- MAPPO-Cermic achieves 172 on VMAS Passage* and 67.3 on Balance*, versus 159 and 60.3 for MAPPO.
- Qmix-Cermic reaches 0.84 on Transport* versus 0.61 for the best curiosity baseline Qplex-Ices.
- The paper reports new SoTA on 12 of 16 evaluated scenarios.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Cermic intrinsic-curiosity calibration to MAPPO | MAPPO · VMAS Flocki* | mean episodic reward: -0.37 -> 1.06 | Helped |
| add Cermic intrinsic-curiosity calibration to MAPPO | MAPPO · VMAS Passage* | mean episodic reward: 159 -> 172 | Helped |
| add Cermic intrinsic-curiosity calibration to MAPPO | MAPPO · VMAS Balance* | mean episodic reward: 60.3 -> 67.3 | Helped |
| add Cermic intrinsic-curiosity calibration to MAPPO | MAPPO · MeltingPot CleanUp | mean episodic return: 74.2 -> 78.3 | Helped |
| add Cermic intrinsic-curiosity calibration to MAPPO | MAPPO · MeltingPot ChiGam | mean episodic return: 8.44 -> 10.03 | Helped |
| add Cermic intrinsic-curiosity calibration to QMIX | QMIX · SMACv2 pro5v5* | mean test win-rate: 0.60 -> 0.73 | Helped |
| add Cermic intrinsic-curiosity calibration to MAPPO | MAPPO · SMACv2 zer5v5* | mean test win-rate: 0.44 -> 0.48 | No clear effect |
| remove exploration loss (ablation) | MAPPO-Cermic · VMAS Flocki* | mean episodic reward: 1.06 ($\pm$0.12) -> 0.82 ($\pm$0.15) | Hurt |
| remove exploration loss (ablation) | MAPPO-Cermic · MeltingPot CleanUp | mean episodic return: 78.1 ($\pm$0.82) -> 75.4 ($\pm$1.05) | Hurt |
| remove exploitation losses (ablation) | MAPPO-Cermic · VMAS Flocki* | mean episodic reward: 1.06 ($\pm$0.12) -> 0.72 ($\pm$0.48) | Hurt |
| remove exploitation losses (ablation) | MAPPO-Cermic · MeltingPot CleanUp | mean episodic return: 78.1 ($\pm$0.82) -> 71.5 ($\pm$1.59) | Hurt |
| set coverage level α to 0.2 rather than 1.0 (ablation) | MAPPO-Cermic · VMAS Flocki* | mean episodic reward: 0.80 ($\pm$0.07) -> 1.06 ($\pm$0.12) | Helped |
Surprising findings
- Curiosity-driven baselines such as MAPPO-DB decline in per-agent contribution as agent count rises, while CERMIC does not.
Limitations
- Learning intention models from local observations under partial observability is hard, causing slower initial sample efficiency.
- One-step latent transition model and graph construction may not scale to higher-dimensional or multi-modal MAS tasks.
Experimental setup
Datasets: VMAS MeltingPot SMACv2
Benchmarks: VMAS MeltingPot SMACv2
Metrics: mean episodic reward mean episodic return mean test win-rate
Baselines: MAPPO QMIX MAPPO-epsilon greedy CPM QMIX-SPECTra MAPPO-DB MACE ICES (QPLEX-Ices)
Builds on: Information Bottleneck MAPPO QMIX LSVI-UCB Bayesian Surprise
Closest papers by meaning
- Curiosity-driven Exploration by Self-supervised Prediction (ICML 2017 · similarity 0.90)
- Episodic Multi-agent Reinforcement Learning with Curiosity-driven Exploration (NeurIPS 2021 · similarity 0.89)
- Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learning (ICML 2024 · similarity 0.89)
- Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.88)
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Behavioral Exploration: Learning to Explore via In-Context Adaptation (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.