Research map / Multi-agent systems and games / Cooperative multi-agent RL
MisoDICE: Multi-Agent Imitation from Mixed-Quality Demonstrations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-stage method for offline multi-agent imitation from unlabeled mixed-quality demonstrations: LLM preferences refined by O-MAPL to label trajectories, then MisoDICE, a value-decomposed DICE objective with linear mixing for convex policy optimization.
Problem
Offline cooperative multi-agent imitation learning when demonstrations are unlabeled and of mixed quality (expert and suboptimal), with expert data scarce.
Approach
Label trajectories by GPT-4o preferences refined through O-MAPL reward recovery, then extend single-agent DICE to multi-agent with a linear value-mixing network so the objective is convex and local policies are consistent with the global weighted BC policy.
Key results
- On SMACv2 (Protoss 5v5), MisoDICE achieves 12.4 ± 0.5 vs 11.7 ± 0.5 for best BC variant and 11.6 ± 0.3 for MARL-SL; MisoDICE is best in most of the 20 SMACv2 maps shown.
- On SMACv2 Terran 20_vs_23, MisoDICE ties VDN at 5.6, the only shown case without a clear win.
- Ablation over top-k expert trajectories shows performance peaks around k=200 and is robust across k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MisoDICE instead of the strongest listed baseline | Offline multi-agent imitation on SMACv2 · SMACv2 tasks in Table 1 | average return: 16.4 ± 1.3 vs 14.6 ± 1.0; 7.3 ± 0.1 vs 6.8 ± 0.1; 8.7 ± 0.2 vs 8.2 ± 0.2 | Helped |
| Replace MisoDICE's learnable mixing with a standard VDN mixer (ablation) | MisoDICE · SMACv2 | average return: 9.2 ± 0.6 vs 8.2 ± 0.5 (Terran 20_vs_20) | Hurt |
| Select a moderate number of top-ranked expert trajectories rather than too few or too many (ablation) | MisoDICE · SMACv2 | average return | Mixed |
Surprising findings
- Authors report LLMs struggle to give meaningful preference feedback in MaMuJoCo, so a rule-based oracle is used there instead.
Limitations
- Limited to cooperative settings
- Reliance on LLM labeling fails in MaMuJoCo where task semantics are hard for LLMs
- Linear mixer is less expressive than nonlinear mixing networks
Experimental setup
Datasets: O-MAPL offline datasets (SMACv1, SMACv2, MaMuJoCo)
Benchmarks: SMACv1 SMACv2
Metrics: average return
Baselines: MA-BC OMAPL INDD MARL-SL VDN
Builds on: DemoDICE O-MAPL QMIX VDN IQ-Learn
Closest papers by meaning
- Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Games (NeurIPS 2024 · similarity 0.87)
- GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation (AAAI 2024 · similarity 0.87)
- Inverse Factorized Soft Q-Learning for Cooperative Multi-agent Imitation Learning (NeurIPS 2024 · similarity 0.87)
- Multi-Agent Imitation Learning: Value is Easy, Regret is Hard (NeurIPS 2024 · similarity 0.86)
- Learning Strategy Representation for Imitation Learning in Multi-Agent Games (AAAI 2025 · similarity 0.86)
- Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.