Research map / Multi-agent systems and games / Cooperative multi-agent RL
Inverse Factorized Soft Q-Learning for Cooperative Multi-agent Imitation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MIFQ, a non-adversarial multi-agent inverse soft-Q imitation learning method that factorizes local Q-values with monotonic hypernetwork-generated mixing networks, and proves convexity of the IQ-Learn objective in the local Q-values under convex non-decreasing mixers.
Problem
Imitation learning in cooperative multi-agent settings with high-dimensional states/actions and inter-agent dependencies; adversarial IL methods (MAGAIL, MAAIRL) are unstable, and naive multi-agent IQ-Learn is not centralized-aware.
Approach
Extend single-agent IQ-Learn to multi-agent by factorizing local soft-Q values and inverse-Bellman local rewards and state values through QMIX-style mixing networks with state-conditioned hypernetworks and non-negative weights, yielding a convex, non-adversarial, centralized-training decentralized-execution objective.
Key results
- With 4096 demonstrations, MIFQ reaches 63% win rate on protoss 5vs5 SMACv2, versus 41% for MAGAIL, 46% for MASQIL, and 87% for the expert.
- On Gold Miner hard, MIFQ achieves 25% win rate versus 20% for MAGAIL and 18% for MASQIL at 4096 demonstrations.
- MIFQ has the best MPE reward on simple_reference (-23.0) and simple_spread (-23.3) but is worse than BC on speaker_listener (-31.3 vs -28.6 for BC at 128 demonstrations; BC is -28.6 in the table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MIFQ instead of the strongest listed baseline | SMACv2 · SMACv2 Protoss 5vs5, Protoss 10vs10, Terran 5vs5 and 10vs10, Zerg 5vs5 and 10vs10 | win rate: 63% vs 46%; 62% vs 38%; 58% vs 29%; 54% vs 33%; 49% vs 21%; 51% vs 33% | Helped |
| use MIFQ instead of the strongest listed baseline | Gold Miner · Gold Miner easy, medium and hard | win rate: 45% vs 34%; 36% vs 25%; 25% vs 20% | Helped |
| use MIFQ instead of the strongest listed baseline | MPE · MPE reference and spread | reward: -23.0 vs -23.2; -23.3 vs -24.4 | Helped |
| use MIFQ instead of BC | MPE · MPE speaker_listener | reward: -31.3 vs -28.6 | Hurt |
Surprising findings
- Behavior cloning performs well on MPE tasks, which the authors attribute to MPE being deterministic.
- MIFQ does not reach expert performance and all IL baselines degrade sharply on the largest SMACv2 settings.
Limitations
- MIFQ does not reach expert-level performance on the evaluated tasks.
- Baselines and MIFQ struggle on very large-scale SMACv2 settings.
- Convexity proof covers mixing network structure but the deep Q-network parameterization is non-convex in parameters, so convexity holds only in the Q-function space.
Experimental setup
Datasets: SMACv2 expert demonstrations (MAPPO-trained experts) Gold Miner expert demonstrations MPE expert demonstrations
Benchmarks: SMACv2 Gold Miner MPE (simple_reference, simple_spread, simple_speaker_listener)
Metrics: win rate average reward
Baselines: BC Independent IQ-Learn (IIQ) IQVDN MASQIL MAGAIL
Builds on: IQ-Learn QMIX MAPPO SQIL VDN
Closest papers by meaning
- Multi-Agent Imitation by Learning and Sampling from Factorized Soft Q-Function (NeurIPS 2025 · similarity 0.92)
- IQ-Learn: Inverse soft-Q Learning for Imitation (NeurIPS 2021 · similarity 0.91)
- Regularized Softmax Deep Multi-Agent Q-Learning (NeurIPS 2021 · similarity 0.90)
- Reward-free World Models for Online Imitation Learning (ICML 2025 · similarity 0.89)
- Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Games (NeurIPS 2024 · similarity 0.88)
- MisoDICE: Multi-Agent Imitation from Mixed-Quality Demonstrations (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.