Research map / Multi-agent systems and games / Cooperative multi-agent RL
ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ComaDICE, an offline cooperative MARL method that regularizes the stationary distribution (DICE-style) with a convex, non-negatively mixed value decomposition and extracts local policies via weighted behavioral cloning.
Problem
Offline cooperative multi-agent RL suffers from distribution shift over large joint state-action spaces; existing methods regularize Q-functions or policies but not stationary distributions.
Approach
Applies DICE-style f-divergence regularization on joint stationary distributions under CTDE by decomposing global value and advantage functions with non-negative mixing networks (proved convex in local values), then extracts local policies via per-agent weighted behavioral cloning using the global occupancy ratio.
Key results
- On SMACv2 Protoss 5_vs_5, ComaDICE reaches 46.2±6.1 winrate versus 33.1±5.4 for OMIGA, and it leads on most SMACv2 instances.
- On MaMujoco Hopper expert, ComaDICE gets 2827.7±62.9 return versus 859.6±709.5 for OMIGA.
- Reported ComaDICE alpha=10 performs best in the alpha sensitivity study.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ComaDICE instead of the strongest listed baseline | Offline MARL on MaMuJoCo · Hopper expert | average return: 859.6±709.5 -> 2827.7±62.9 | Helped |
| Use ComaDICE instead of the strongest listed baseline | Offline MARL on MaMuJoCo · Hopper medium | average return: 1189.3±544.3 -> 822.6±66.2 | Hurt |
| Use ComaDICE instead of the strongest listed baseline | Offline MARL on MaMuJoCo · Ant tasks | average return | Helped |
| Use ComaDICE instead of the strongest listed baseline | Offline MARL on MaMuJoCo · HalfCheetah medium | average return: 3608.1±237.4 -> 2664.7±54.2 | Hurt |
| Use ComaDICE instead of the strongest listed baseline | Offline MARL on MaMuJoCo · HalfCheetah expert, m-replay, and m-expert | average return | Helped |
| Use ComaDICE instead of the strongest listed baseline | Offline MARL on SMACv2 · Protoss tasks | win rate | Helped |
| Use ComaDICE instead of the strongest listed baseline | Offline MARL on SMACv2 · Terran tasks | win rate | Helped |
| Use ComaDICE instead of the strongest listed baseline | Offline MARL on SMACv2 · Zerg tasks | win rate | Mixed |
| Set regularization parameter α around 10 rather than other tested values (ablation) | ComaDICE · SMACv2 and MaMuJoCo | win rate and returns | Helped |
| Use a 1-layer linear mixer instead of a 2-layer mixer (ablation) | ComaDICE · Offline MARL experiments | performance stability | Helped |
Surprising findings
- The linear (1-layer) mixing network gave more stable results than the 2-layer mixer in offline settings, despite 2-layer mixers being strong in online MARL.
- ComaDICE is not best on every task; for example OMIGA wins on MaMujoco Hopper medium and SMACv2 Zerg 20_vs_20 is won by OMAR.
Limitations
- Limited to cooperative settings; cooperative-competitive settings not addressed.
- Performance depends heavily on the quality of the behavior policy.
- Requires large amounts of data to reach good performance; sample efficiency is not addressed.
Experimental setup
Datasets: SMACv1 offline datasets (MAPPO-generated) SMACv2 offline datasets (MAPPO-generated, 1,000 trajectories per instance) MaMujoco offline datasets (HAPPO-generated)
Benchmarks: SMACv1 SMACv2 MaMujoco
Metrics: winrate average return
Baselines: BC BCQ CQL ICQ OMAR OMIGA
Builds on: DICE OMIGA CQL QMIX
Closest papers by meaning
- AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation (NeurIPS 2023 · similarity 0.93)
- Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization (AAAI 2025 · similarity 0.92)
- Cooperative Policy Agreement: Learning Diverse Policy for Offline MARL (AAAI 2025 · similarity 0.91)
- Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation (NeurIPS 2024 · similarity 0.91)
- Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rank (ICLR 2025 · similarity 0.89)
- Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.