Research map / Multi-agent systems and games / Cooperative multi-agent RL
Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IIE, which uses a Decision-Transformer-style imagination model with prompts (timestep-to-go, return-to-go, influence value, one-shot demo) to imagine trajectories to critical states, then initializes SMAC agents there to explore.
Problem
Multi-agent exploration in long-horizon, sparse-reward coordination tasks is inefficient because agents rarely obtain the specific joint action sequences needed to reach successful, under-explored interaction states.
Approach
Use a prompt-conditioned transformer to imagine how agents reach high-influence interaction states from the initial state, initialize the simulator at the imagined last state, and explore from there; the imagined and explored trajectories are stitched for policy training under a QMIX-style value decomposition.
Key results
- IIE outperforms multi-agent exploration baselines on dense-reward SMAC and SMACv2 maps, with best final performance across scenarios.
- On sparse-reward SMAC, IIE remains robust while QMIX and MAPPO fail to solve the tasks, per the paper's reported results.
- IIE outperforms behavior cloning, GC-policy, CVAE-GAN, and classifier-guided diffusion as initialization sources across all tested tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IIE instead of the compared MARL exploration methods | MARL policies · SMAC dense-reward tasks | performance | Helped |
| Use IIE instead of the compared methods | MARL policies · SMACv2 | performance | Helped |
| Use IIE instead of the compared exploration methods | MARL policies · SMAC sparse-reward tasks | performance | Helped |
| Use IIE instead of behavior cloning, GC-policy, CVAE-GAN, or classifier-guided diffusion | trajectory returning for exploration · SMAC tasks | performance | Helped |
| Remove the return-to-go prompt component (ablation) | IIE imagination model · SMAC MMM2, 6h_vs_8z, and corridor | performance | Hurt |
| Remove the influence-value prompt component (ablation) | IIE imagination model · SMAC MMM2, 6h_vs_8z, and corridor | performance | Hurt |
| Add the most related one-shot trajectory demonstration (ablation) | IIE imagination model · SMAC tasks, especially long-horizon tasks | performance | Helped |
| Add the return-to-go prompt component (ablation) | IIE-wo-IV · SMAC MMM2, 6h_vs_8z, and 2c_vs_64zg | performance | Helped |
| Add the return-to-go prompt component (ablation) | IIE-wo-IV · SMAC corridor | performance | Hurt |
| Use return-to-go alone as the prompt (ablation) | IIE imagination model · SMAC MMM2, 6h_vs_8z, and corridor | performance | Hurt |
| Use influence value alone as the prompt (ablation) | IIE imagination model · SMAC tasks | performance | Hurt |
| Remove both influence value and the one-shot trajectory demonstration, retaining RTG (ablation) | IIE imagination model (RTG variant) · SMAC tasks | performance | Hurt |
Surprising findings
- CVAE-GAN shows no positive results, attributed to mode collapse, and behavior cloning performs worst among returning methods.
- Ablation finds IIE-wo-IV performs better than IIE on the corridor map, suggesting RTG may reduce diversity there.
Limitations
- Evaluation is confined to StarCraft II; generality to other multi-agent benchmarks is untested.
- Pretraining the imagination model delays learning, so early learning is slower than EMC on complex maps like 3s5z_vs_3s6z and corridor.
- The simulator reset relies on StarCraft II debug commands and a health-scaling hack, limiting portability.
Experimental setup
Datasets: SMAC SMACv2
Benchmarks: SMAC SMACv2
Metrics: win rate mean health of allies and enemies relative distance
Baselines: CW-QMIX QPLEX MAVEN EMC RODE QMIX MAPPO LIIR MASER behavior cloning GC-policy CVAE-GAN
Builds on: Decision Transformer QMIX Go-Explore MAVEN EMC
Closest papers by meaning
- Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learning (ICML 2024 · similarity 0.88)
- Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibration (NeurIPS 2025 · similarity 0.88)
- Settling Decentralized Multi-Agent Coordinated Exploration by Novelty Sharing (AAAI 2024 · similarity 0.87)
- Grounded Answers for Multi-agent Decision-making Problem through Generative World Model (NeurIPS 2024 · similarity 0.87)
- Learning World Models for Unconstrained Goal Navigation (NeurIPS 2024 · similarity 0.87)
- Off-policy Reinforcement Learning with Model-based Exploration Augmentation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.