Research map / Multi-agent systems and games / Cooperative multi-agent RL
O-MAPL: Offline Multi-agent Preference Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes O-MAPL, an end-to-end offline multi-agent preference-based RL method that learns soft Q-functions directly from trajectory preferences via linear value factorization and weighted behavior cloning for local policies.
Problem
Recovering joint reward functions and policies in cooperative multi-agent RL from offline pairwise trajectory preferences, where existing two-phase methods (reward learning then MARL) suffer from large data needs and misalignment.
Approach
Exploit the one-to-one mapping between reward and soft Q-function in MaxEnt RL to learn a soft Q-function directly from preferences, factorize it with a single-layer linear mixing network under CTDE (convex objective, global-local consistency), and extract decentralized policies via a local weighted behavior cloning objective.
Key results
- On SMACv1 2c_vs_64zg with rule-based labels, O-MAPL reaches 74.4% win rate versus 71.1% for IPL-VDN.
- On corridor, O-MAPL achieves 93.2% win rate (rule-based) and 94.5% (LLM-based), matching or exceeding baselines.
- On MaMujoco Hopper-v2, O-MAPL achieves 1114.4 return versus 890.0 for the best baseline SL-MARL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use O-MAPL instead of IPL-VDN | SMACv1 2c_vs_64zg with rule-based preferences · SMACv1 2c_vs_64zg | win rate (%): 71.1±22.0 -> 74.4±24.7 | Helped |
| Use O-MAPL instead of IPL-VDN | SMACv1 corridor with rule-based preferences · SMACv1 corridor | win rate (%): 93.9±11.6 -> 93.2±13.5 | Hurt |
| Use O-MAPL instead of IPL-VDN | SMACv1 corridor with LLM-based preferences · SMACv1 corridor | win rate (%): 94.5±12.5 -> 94.5±11.2 | No clear effect |
| Use O-MAPL instead of IPL-VDN | MaMujoco Hopper-v2 · Hopper-v2 | return: 890.0 ± 88.7 -> 1114.4 ± 154.1 | Helped |
| Use O-MAPL instead of IPL-VDN | MaMujoco Ant-v2 · Ant-v2 | return: 1376.1 ± 142.0 -> 1406.4 ± 163.7 | Helped |
| Use O-MAPL instead of IPL-VDN | MaMujoco HalfCheetah-v2 · HalfCheetah-v2 | return: 4287.5 ± 273.1 -> 4382.0 ± 189.7 | Helped |
| Use LLM-generated instead of rule-based preference data | O-MAPL on SMAC tasks · SMACv1 and SMACv2 | win rate (%) | Mixed |
Surprising findings
- LLM-based (GPT-4o) preference labels yield higher win rates than rule-based labels on most SMAC tasks.
Limitations
- Focuses only on cooperative settings; mixed cooperative-competitive settings are not addressed.
- Depends on large numbers of preference-based demonstrations for good policies.
- Sample efficiency remains a challenge when real human feedback is required.
Experimental setup
Datasets: SMACv1 offline datasets (MAPPO-generated) SMACv2 offline dataset (ComaDICE) MaMujoco offline datasets (HAPPO-generated, OMIGA)
Benchmarks: SMACv1 SMACv2 MaMujoco
Metrics: win rate return
Baselines: BC IIPL IPL-VDN SL-MARL
Builds on: Inverse Preference Learning (IPL) OMIGA Extreme Q-Learning VDN MaxEnt RL
Closest papers by meaning
- Decoding Global Preferences: Temporal and Cooperative Dependency Modeling in Multi-Agent Preference-Based Reinforcement Learning (AAAI 2024 · similarity 0.94)
- Preference Elicitation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.89)
- Provable Reward-Agnostic Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.89)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.