Research map / Multi-agent systems and games / Cooperative multi-agent RL
Cooperative Policy Agreement: Learning Diverse Policy for Offline MARL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cooperative Policy Agreement (CPA), an autoregressive offline MARL method that avoids policy mismatch on diverse datasets and distills it into CTDE-compliant decentralized policies via a state-conditional action pool and CVAE-based agreement key.
Problem
Offline MARL on diverse datasets with multiple optimal joint policies suffers from policy mismatch, where individually optimal actions combine into suboptimal joint actions, and existing methods also fail to produce diverse joint policies under CTDE.
Approach
Train an autoregressive joint policy (agents condition on predecessors' actions) on mixed offline data to eliminate mismatch, then distill it into non-autoregressive decentralized policies using a shared state-conditional action pool (VQ) and a CVAE-based agreement key that agents set from a common seed without communication.
Key results
- On Aloha (poor/medium/good), AR reaches 74.43/74.85/76.12 and PA 73.12/73.25/75.34, versus best baseline AlberDICE at about 49.
- On XOR, AR and PA reach 1.00 on all datasets while BC, BCQ, CQL and MAICQ are mostly below 1.0.
- PA with the policy agreement mechanism performs nearly identically to the autoregressive AR model, and both keep performance while entropy increases in diversity-performance plots.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the policy-agreement method instead of BC | Offline MARL · Sensor, Poor | return: 15.91 ± 1.75 -> 28.39 ± 0.72 | Helped |
| Use the policy-agreement method instead of BC | Offline MARL · Sensor, Medium | return: 20.20 ± 5.38 -> 28.71 ± 0.32 | Helped |
| Use the policy-agreement method instead of BC | Offline MARL · Sensor, Good | return: 28.32 ± 0.28 -> 29.22 ± 0.72 | Helped |
| Use the policy-agreement method instead of AlberDICE | Offline MARL · Aloha, Poor | return: 49.24 ± 0.55 -> 73.12 ± 0.51 | Helped |
| Use the policy-agreement method instead of AlberDICE | Offline MARL · Aloha, Medium | return: 49.42 ± 0.90 -> 73.25 ± 0.63 | Helped |
| Use the policy-agreement method instead of AlberDICE | Offline MARL · Aloha, Good | return: 48.31 ± 1.69 -> 75.34 ± 0.84 | Helped |
| Use the policy-agreement method instead of AlberDICE | Offline MARL · Bridge, Poor | return: -1.40 ± 0.26 -> -1.27 ± 0.02 | Helped |
| Use the policy-agreement method instead of AlberDICE | Offline MARL · Bridge, Medium | return: -1.29 ± 0.00 -> -1.24 ± 0.00 | Helped |
| Use the policy-agreement method instead of AlberDICE | Offline MARL · Bridge, Good | return: -1.27 ± 0.03 -> -1.24 ± 0.00 | Helped |
| Use the policy-agreement method instead of AlberDICE | Offline MARL · XOR | return: 1.00 ± 0.00 -> 1.00 ± 0.00 | No clear effect |
| Use the policy-agreement method instead of AlberDICE | Offline MARL · Permutation | return: 1.00 ± 0.00 -> 1.00 ± 0.00 | No clear effect |
| Increase policy diversity by varying the threshold β | AR and PA · Permutation and Aloha | return | Mixed |
Surprising findings
- Many baselines (BC, BCQ, CQL, MAICQ, AlberDICE) show no correlation between dataset quality and final performance, which the authors attribute to policy mismatch.
Limitations
- Restricted to MMDP setting; not applicable to Dec-POMDP without observation exchange.
- Autoregressive model requires O(N) communication, so the policy agreement mechanism is needed for decentralized execution.
- Policy agreement may fail when partial observations diverge significantly from the global state.
Experimental setup
Datasets: XOR offline datasets Permutation offline datasets Bridge offline datasets Sensor offline datasets Aloha offline datasets
Benchmarks: XOR Permutation Bridge Sensor Aloha
Metrics: mean return over 5 seeds standard error policy average entropy
Baselines: BC BCQ CQL MAICQ AlberDICE
Builds on: MAICQ ICQ Multi-agent MDP CVAE VQ-VAE
Closest papers by meaning
- ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization (ICLR 2025 · similarity 0.91)
- Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization (AAAI 2025 · similarity 0.91)
- Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.90)
- Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.89)
- Learning Joint Behaviors with Large Variations (AAAI 2025 · similarity 0.89)
- Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.