Research map / Multi-agent systems and games / Cooperative multi-agent RL
Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Audits offline MARL papers for inconsistent baselines and evaluation, then shows simple well-implemented baselines (BC, IQL+CQL, IDDPG+BC, MADDPG+CQL) match or beat published SOTA on 35 of 47 datasets, and releases standardized baselines and evaluation guidelines.
Problem
Offline multi-agent RL research suffers from inconsistent baseline naming/implementations, mismatched scenarios, and inconsistent evaluation protocols, making it hard to judge real progress.
Approach
Re-evaluate prior offline MARL claims by training simple standardized baselines on the original authors' datasets and comparing against values reported in those papers, then propose evaluation guidelines and an OG-MARL-based standardized dataset/baseline release.
Key results
- Simple baselines match or surpass the purported SOTA on 35 of 47 datasets (about 75%).
- On the 8m Medium SMACv1 dataset, IQL+CQL reaches 19.10 final return versus BC's 10.65, while performance-over-training shows BC can be preferred at later steps.
- On MAMuJoCo 2x3 HalfCheetah medium-replay, IDDPG+BC reaches 719.03 versus 668.97 for MADDPG+CQL in the OMAR table, illustrating baselines competitive with published methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the paper's simple baseline implementations rather than the reported literature methods | Offline MARL across the tested datasets · 47 datasets across SMAC, MPE, and MAMuJoCo | performance relative to literature SOTA: 35 out of the 47 datasets | Mixed |
| Train IDDPG+BC instead of the reported algorithms | MPE cooperative navigation medium · MPE CN medium | normalised episode return: 65.00$\pm$10.20 -> 102.02$\pm$10.68 | Helped |
| Train IDDPG+BC instead of the reported algorithms | MAMuJoCo 2x3 HalfCheetah · medium-replay | normalised episode return: 59.50$\pm$8.20 -> 719.03$\pm$37.31 | Helped |
| Train IDDPG+BC instead of the reported algorithms | MAMuJoCo 3x1 Hopper · expert | episode return: 859.63$\pm$709.47 -> 3553.13$\pm$107.21 | Helped |
| Train MADDPG+CQL instead of OMIGA | MAMuJoCo 2x4 Ant · medium-expert | episode return: 1720.33$\pm$110.63 -> 1972.17$\pm$129.61 | Helped |
| Train IQL+CQL instead of OMIGA | SMAC 5m vs 6m · good | episode return: 8.25$\pm$0.37 -> 12.36$\pm$1.09 | Helped |
| Train IQL+CQL instead of OMIGA | SMAC corridor · good | episode return: 15.88$\pm$0.89 -> 19.06$\pm$0.81 | Helped |
| Train IQL+CQL instead of CFCQL | SMAC 2s3z · mixed | win rate: 0.84$\pm$0.09 -> 0.97$\pm$0.03 | Helped |
| Train IQL+CQL instead of CFCQL | SMAC 3s vs 5z · medium-replay | win rate: 0.12$\pm$0.04 -> 0.26$\pm$0.16 | Helped |
| Increase training from 25k to 50k steps | QMIX+CQL versus behaviour cloning · SMACv1 8m Medium | episode return: 25k -> 50k steps | Mixed |
| Change from QMIX+CQL to MADDPG+CQL | Two plausible MACQL implementations · SMACv1 8m and 5m_vs_6m | episode return | Mixed |
| Tune CQL sigma by scenario (0.1 for Ant, 0.2 for Hopper, 0.3 for HalfCheetah) | MADDPG+CQL · MAMuJoCo | episode return | Helped |
Surprising findings
- Well-implemented simple baselines often substantially outperform sophisticated published offline MARL algorithms.
- The ranking between algorithms can flip depending on the training budget (e.g., stopping at 25k vs 50k steps), so training budget choice can drive conclusions.
Limitations
- Results depend on the original authors' datasets and modified environment configurations, which the paper does not advocate for future work.
- Could not access some datasets (e.g., MPE PP and WD, several SMAC datasets), limiting coverage.
- Pre-trained models were not saved, and the study covers only a few algorithms and environments.
Experimental setup
Datasets: SMACv1 datasets from prior work MAMuJoCo datasets from prior work MPE datasets from prior work OG-MARL datasets
Benchmarks: SMACv1 MAMuJoCo MPE
Metrics: final episode return average and maximum episode return normalised score heteroscedastic two-sided t-test interquartile mean performance profiles
Baselines: MAICQ OMAR MADT CFCQL OMIGA MABCQ MACQL OMAC
Builds on: conservative q-learning maddpg td3+bc og-marl
Closest papers by meaning
- Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization (AAAI 2025 · similarity 0.92)
- ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization (ICLR 2025 · similarity 0.91)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Foundations of Multi-Agent Learning in Dynamic Environments: Where Reinforcement Learning Meets Strategic Decision-Making (AAAI 2025 · similarity 0.90)
- Forecasting in Offline Reinforcement Learning for Non-stationary Environments (NeurIPS 2025 · similarity 0.90)
- Scaling Offline Evaluation of Reinforcement Learning Agents through Abstraction (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.