Research map / Multi-agent systems and games / Cooperative multi-agent RL
Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InSPO, an in-sample sequential policy optimization method for offline MARL that adds policy entropy to a behavior-regularized Markov game so it converges to quantal response equilibrium and escapes local optima.
Problem
Offline multi-agent RL suffers from out-of-distribution joint actions, uncoordinated behaviors, and premature convergence to sub-optimal equilibria when the behavior dataset concentrates on local optima.
Approach
Sequentially update each agent's policy with a closed-form in-sample objective derived from reverse-KL behavior regularization, add a policy-entropy term to encourage exploration of low-probability dataset actions, and use importance resampling to correct for teammates' updated policies; theory shows monotonic improvement and QRE convergence.
Key results
- On StarCraft II, InSPO achieves average winning rate 0.55 across four maps and four datasets, the highest among BC (0.38), OMAR (0.21), CFCQL (0.54), and OMIGA (0.39).
- On the imbalanced M-NE game, only InSPO reaches the global optimum with return 20.00 versus 5.00 for the other offline baselines.
- On the XOR game, InSPO and AlberDICE reach optimal return 1.00 on all three datasets, while OMAR, CFCQL, and OMIGA fail on at least one.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply InSPO's sequential in-sample optimization | offline MARL · XOR game datasets (a), (b), and (c) | averaged test return | Helped |
| apply InSPO to a dataset biased toward a local optimum | offline MARL · M-NE game, imbalanced | averaged test return: 5.00±0.00 -> 20.00±0.00 | Helped |
| apply sequential in-sample optimization | offline MARL · Bridge, optimal and mixed | averaged test return | Helped |
| add policy entropy (ablation) | InSPO · M-NE game, imbalanced | convergence to global optimum | Helped |
| use sequential rather than simultaneous policy updates (ablation) | InSPO · XOR game (b) | optimal policy learning | Helped |
| use auto-tuned α rather than fixed α=10 (ablation) | InSPO · StarCraft II 6h_vs_8z, expert | winning rate: 0.69± 0.05 -> 0.74 | Helped |
| use auto-tuned α rather than fixed α=5 (ablation) | InSPO · StarCraft II 6h_vs_8z, mixed | winning rate: 0.54± 0.06 -> 0.60 | Helped |
| vary the agent update order among random, fixed, and semi-greedy (ablation) | InSPO · StarCraft II 5m_vs_6m and 6h_vs_8z | winning rate | No clear effect |
Surprising findings
- Value decomposition methods (OMIGA, CFCQL) can converge to out-of-distribution joint actions like (B,B) in the XOR game, and OMAR fails there too.
- On imbalanced M-NE data, most offline baselines converge to the local optimum (A,A) determined by the behavior policy.
Limitations
- Sequential updates increase training time relative to simultaneous-update methods.
- Auto-tuned alpha and update order are heuristic; the paper finds update order has limited effect but the optimal order is unresolved.
- QRE convergence proof is stated in the tabular setting, while the practical implementation uses function approximation.
Experimental setup
Datasets: StarCraft II micromanagement offline datasets (medium, expert, medium-replay, mixed) Bridge offline datasets (optimal, mixed) XOR game datasets M-NE game datasets
Benchmarks: XOR game M-NE game Bridge StarCraft II micromanagement (2s3z, 3s_vs_5z, 5m_vs_6m, 6h_vs_8z)
Metrics: averaged test return test winning rate
Baselines: BC OMAR AlberDICE CFCQL OMIGA
Builds on: CFCQL OMAR AlberDICE OMIGA CQL
Closest papers by meaning
- Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation (NeurIPS 2024 · similarity 0.92)
- ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization (ICLR 2025 · similarity 0.92)
- Cooperative Policy Agreement: Learning Diverse Policy for Offline MARL (AAAI 2025 · similarity 0.91)
- Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rank (ICLR 2025 · similarity 0.90)
- A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective (NeurIPS 2024 · similarity 0.89)
- Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.