Research map / Multi-agent systems and games / Cooperative multi-agent RL
Efficient Multi-agent Reinforcement Learning by Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAZero, a MuZero-style model-based MARL algorithm that combines a centralized-value, individual-dynamics learned model with sampled MCTS, plus OS(λ) optimistic search and AWPO advantage-weighted policy loss, evaluated on SMAC.
Problem
Model-based multi-agent RL with planning is underexplored; existing single-agent MuZero-style models ignore multi-agent biases (near-independence, parameter sharing) and planning in exponentially large joint action spaces is inefficient and underestimates values.
Approach
A centralized-value, individual-dynamic latent model with parameter sharing and attention-based communication, combined with sampled MCTS; OS(λ) replaces mean value estimates with optimistic depth-weighted quantile estimates for deterministic models, and AWPO weights behavior-cloning loss by exponentiated optimistic advantages.
Key results
- On SMAC, MAZero outperforms model-free baselines (QMIX, QPLEX, RODE, CDS, MAPPO) across eight scenarios in sample efficiency and matches or beats MAMBA in sample and computational efficiency.
- Ablation: removing OS(λ) and AWPO reduces performance, and the flattened single-agent model only solves the simplest map (2m_vs_1z).
- On LunarLander (400 discretized actions) and Walker2D (~280k discretized actions), MAZero with OS(λ)+AWPO beats the ablated variant; e.g., Walker2D 1M steps 5189±382 vs 4266±509.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AWPO instead of behavior cloning | MAZero policy optimization · 100-armed bandit | convergence speed | Helped |
| use OS(λ) and AWPO rather than removing both (ablation) | MAZero · LunarLander | return: 104.0±87.5 -> 184.6±22.8 | Helped |
| use OS(λ) and AWPO rather than removing both (ablation) | MAZero · Walker2D | return: 2302 ± 472 -> 3424 ± 246 | Helped |
| use MCTS during evaluation (ablation) | MAZero · SMAC 2m_vs_1z, 50k env steps | win rate: 1.0 ± 0.0 -> 1.0 ± 0.0 | No clear effect |
| use MCTS during evaluation (ablation) | MAZero · SMAC 2s_vs_1sc, 100k env steps | win rate: 0.623 ± 0.185 -> 0.948 ± 0.072 | Helped |
| use MCTS during evaluation (ablation) | MAZero · SMAC average performance | performance ratio: 90.1 ± 11.3% -> 100% | Helped |
| use the Adam optimizer rather than SGD (ablation) | MAZero · SMAC | performance and training stability | Helped |
| increase the MCTS sampled scale from K=5,N=50 to K=10,N=100 (ablation) | MAZero · SMAC 8m and 5m_vs_6m | performance | Helped |
| vary OS(λ) quantile ρ (ablation) | MAZero · SMAC 8m | sample efficiency | No clear effect |
| use λ weighting in OS(λ) to account for model error (ablation) | MAZero · SMAC 8m | sample efficiency | Helped |
| remove communication and parameter sharing from the network structure (ablation) | MAZero · SMAC | algorithm performance | Hurt |
| replace the multi-agent network structure with a flattened single-agent MBRL model (ablation) | MAZero · SMAC | ability to learn winning strategies | Hurt |
Surprising findings
- Agents maintain comparable performance without MCTS planning at evaluation time (average 90.1% of performance with MCTS removed), suggesting the learned policy is largely distilled from search.
- Adam outperforms SGD for the MuZero-style training, contrary to the SGD default in most MuZero-based algorithms.
Limitations
- Experiments limited to cooperative settings, with evaluation mainly on SMAC; single-agent and GRF results are supplementary.
- Relies on a deterministic environment assumption for OS(λ); performance in stochastic settings is not addressed.
- Large joint action spaces remain a challenge; the authors note the need for action representation to reduce dimensionality.
Experimental setup
Datasets: SMAC Google Research Football LunarLander Walker2D
Benchmarks: SMAC Google Research Football academy_pass_and_shoot_with_keeper LunarLander Walker2D
Metrics: win rate sample efficiency wall-clock run time interquartile mean optimality gap
Baselines: MAMBA QMIX QPLEX RODE CDS MAPPO
Builds on: MuZero Sampled MuZero EfficientZero MAMBA DreamerV2
Closest papers by meaning
- Multiagent Gumbel MuZero: Efficient Planning in Combinatorial Action Spaces (AAAI 2024 · similarity 0.88)
- MALinZero: Efficient Low-Dimensional Search for Mastering Complex Multi-Agent Planning (NeurIPS 2025 · similarity 0.88)
- Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization (AAAI 2025 · similarity 0.88)
- Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective (NeurIPS 2025 · similarity 0.88)
- LNS2+RL: Combining Multi-agent Reinforcement Learning with Large Neighborhood Search in Multi-agent Path Finding (AAAI 2025 · similarity 0.87)
- FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.