Research map / Multi-agent systems and games / Cooperative multi-agent RL
Settling Decentralized Multi-Agent Coordinated Exploration by Novelty Sharing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MACE, a decentralized MARL coordinated exploration method that shares only scalar local novelty and adds a hindsight weighted-mutual-information intrinsic reward on top of IPPO.
Problem
Exploration in decentralized cooperative multi-agent RL under sparse rewards, where agents only see local observations, local novelty is a biased estimate of global novelty, and agents need coordinated exploration with limited communication.
Approach
Agents broadcast a scalar local novelty so each approximates global novelty by summing all agents' novelty, and receive a hindsight intrinsic reward based on weighted mutual information between their action and other agents' future accumulated novelty, which rewards actions that influence others' exploration.
Key results
- MACE achieves the best performance on all three GridWorld tasks (Pass, SecretRoom, MultiRoom), while IPPO+r_loc fails to solve any task.
- In Overcooked and SMAC with sparse rewards, MACE outperforms IPPO+r_loc, IPPO+r_nov, and IPPO+r_hin in learning speed or win rate.
- Weighted mutual information outperforms plain mutual information as the influence measure (MACE-MI is less effective than MACE in all tasks).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add summed local novelty from other agents to approximate global novelty (ablation) | IPPO exploration · GridWorld Pass, SecretRoom, MultiRoom | mean reward | Helped |
| add hindsight-based intrinsic reward for influencing other agents' exploration (ablation) | IPPO with novelty-based reward · GridWorld Pass, SecretRoom, MultiRoom | mean reward | Helped |
| replace weighted mutual information reward with the logarithmic mutual-information term (ablation) | MACE · GridWorld tasks | mean reward | Hurt |
| replace hindsight reward with other agents' accumulated novelty alone (ablation) | MACE · GridWorld tasks | mean reward | Hurt |
| use maximum instead of sum of agents' local novelty (ablation) | IPPO novelty reward · GridWorld Pass | mean reward | Hurt |
| use local novelty only instead of approximated global novelty (ablation) | IPPO exploration · GridWorld tasks | mean reward | Hurt |
| use scalable hindsight reward based on summed novelty of other agents (ablation) | MACE · GridWorld MultiRoom | mean reward | No clear effect |
| use scalable rather than agent-specific hindsight reward (ablation) | IPPO+r_hin · GridWorld MultiRoom | mean reward | Hurt |
| set posterior-estimation history window w to 1 rather than 10 (ablation) | MACE · GridWorld SecretRoom and MultiRoom | mean reward | Hurt |
| set posterior-estimation history window w to 50 rather than 10 (ablation) | MACE · GridWorld Pass and SecretRoom | mean reward | Hurt |
| set hindsight-reward weight λ to 0.1 or 0.001 rather than 0.01 (ablation) | MACE · GridWorld tasks | mean reward | Hurt |
| use approximated global novelty only, without hindsight reward (ablation) | IPPO · Overcooked Base, Narrow, Large | mean reward | Mixed |
Surprising findings
- MASER, a centralized multi-agent exploration method, performed ineffectively on the GridWorld tasks under default hyperparameters.
- IPPO+r_nov outperforms IPPO+r_hin in Overcooked, suggesting the novelty-based reward may matter more in complex tasks.
Limitations
- Requires a fully-connected communication network to share novelty among agents.
- Hindsight reward needs on-policy samples, so it is currently built on on-policy IPPO and not combined with off-policy decentralized methods.
- Approximation to global novelty via summation still deviates from true global novelty when agents rarely visit states simultaneously.
Experimental setup
Datasets: GridWorld (Pass, SecretRoom, MultiRoom) Overcooked (Base, Narrow, Large) SMAC (2m_vs_1z, 3m, 8m)
Benchmarks: GridWorld Pass GridWorld SecretRoom GridWorld MultiRoom Overcooked Base Overcooked Narrow Overcooked Large SMAC 2m_vs_1z SMAC 3m SMAC 8m
Metrics: mean episode reward win rate
Baselines: IPPO+r_loc IPPO+r_nov IPPO+r_hin MACE-MI MACE-Z MAPPO MASER IPPO
Builds on: IPPO PPO RND pseudo-count MAPPO
Closest papers by meaning
- Randomized Exploration in Cooperative Multi-Agent Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learning (ICML 2024 · similarity 0.88)
- The Horizon Threshold in Cooperative Multi-Agent Reward-Free Exploration (NeurIPS 2026 · similarity 0.88)
- Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learning (AAAI 2025 · similarity 0.88)
- Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Efficient Communication in Multi-Agent Reinforcement Learning with Implicit Consensus Generation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.