Research map / Multi-agent systems and games / Cooperative multi-agent RL
BenchMARL: Benchmarking Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BenchMARL, a PyTorch MARL training library built on TorchRL that standardizes configuration, algorithms, models, and environments for reproducible benchmarking, and reports baseline results of 9 algorithms on 3 VMAS tasks.
Problem
MARL research suffers from fragmented tooling and a reproducibility crisis, lacking a standardized benchmarking library that uses modern single-agent RL implementations.
Approach
A MARL library that reuses TorchRL single-agent RL components with abstractions for experiments, benchmarks, algorithms, tasks, and models, configured via YAML/Hydra and integrated with statistically rigorous reporting tools, supporting agent grouping and vectorized simulation.
Key results
- Benchmarked 9 MARL algorithms on 3 VMAS tasks with 95% stratified bootstrap confidence intervals over 3 seeds.
- Multi-agent actor-critic variants (MASAC, MADDPG, MAPPO) obtained the best overall performance across VMAS tasks.
- Q-learning algorithms (IQL, VDN, QMIX) performed suboptimally versus actor-critic methods, possibly due to discrete action versions of tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-agent actor-critic algorithms with centralized critics | BenchMARL algorithms · VMAS Navigation, Sampling, and Balance | performance | Helped |
| train with Q-learning algorithms instead of actor-critic algorithms | BenchMARL algorithms · VMAS Navigation, Sampling, and Balance | performance | Hurt |
Surprising findings
- Q-learning based methods (IQL, VDN, QMIX) underperform actor-critic methods on continuous multi-robot control tasks.
Limitations
- Experimental evaluation limited to three VMAS tasks with only 3 seeds.
- Q-learning baselines were run on discrete-action versions of continuous tasks, which may impede their performance.
Experimental setup
Datasets: VMAS Navigation VMAS Sampling VMAS Balance
Benchmarks: VMAS Navigation VMAS Sampling VMAS Balance
Metrics: inter-quartile mean (IQM) sample efficiency performance profile aggregate scores
Baselines: MAPPO IPPO MADDPG IDDPG MASAC ISAC QMIX VDN IQL
Builds on: TorchRL Hydra PyMARL RLlib SAC
Closest papers by meaning
- MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning (ICML 2026 · similarity 0.87)
- Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (NeurIPS 2021 · similarity 0.87)
- SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning (NeurIPS 2023 · similarity 0.86)
- MARPO: A Reflective Policy Optimization for Multi-Agent Reinforcement Learning (AAAI 2026 · similarity 0.86)
- Agentick: A Unified Benchmark for General Sequential Decision-Making Agents (NeurIPS 2026 · similarity 0.85)
- TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarios (ICML 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.