Research map / Multi-agent systems and games / Cooperative multi-agent RL
Decoding Global Preferences: Temporal and Cooperative Dependency Modeling in Multi-Agent Preference-Based Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAPT, a cascaded Transformer reward model for multi-agent preference-based RL that captures temporal and cooperative dependencies, then trains MARL agents (MAT) on the learned rewards from scripted-teacher preferences.
Problem
Preference-based RL reward models for multi-agent cooperative tasks fail to capture temporal and cooperative (inter-agent) dependencies, producing homogeneous rewards and poor credit assignment.
Approach
A cascaded Transformer preference model with a cooperative-aware preference attention layer (inter-agent self-attention, per-agent rewards averaged into a collective reward) and a temporal-aware preference attention layer (causal attention producing temporal importance weights), trained with Bradley-Terry cross-entropy on multi-agent preferences.
Key results
- MAPT improves accumulated trajectory rewards by 35% to 189% over baselines on discrete-action SMAC and Football tasks.
- On 3s5z and 6h vs 8z, baselines reach 0% win rate while MAPT reaches 94% and 62%.
- On continuous-action Bi-DexHands and Ma-Mujoco, MAPT gives an average 46% improvement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MAPT instead of the strongest preference-model baseline, MA-LSTM | MAT policy training · 3s5z | accumulated trajectory reward: 14.52±0.49 -> 19.59±0.42 | Helped |
| Use MAPT instead of the strongest preference-model baseline, MA-LSTM | MAT policy training · 6h vs 8z | accumulated trajectory reward: 9.76±0.15 -> 17.05±0.53 | Helped |
| Use MAPT instead of the strongest preference-model baseline, MA-LSTM | MAT policy training · CatchOver | accumulated trajectory reward: 25.34±1.59 -> 25.12±0.64 | No clear effect |
| Use MAPT instead of the strongest preference-model baseline, MA-LSTM | MAT policy training · DoorOpenInward | accumulated trajectory reward: 242.42±33.92 -> 372.60±11.38 | Helped |
| Add the temporal-aware preference attention layer (ablation) | MAPT · OpenIn., CloseOut., 3s5z, 6h vs 8z | accumulated trajectory reward: 23.43±0.95 -> 292.16±11.38; 718.16±22.38 -> 786.70±26.03; 8.32±0.38 -> 19.59±0.42; 13.26±0.31 -> 17.05±0.53 | Helped |
| Add the cooperation-aware preference attention layer (ablation) | MAPT · OpenIn., CloseOut., 3s5z, 6h vs 8z | accumulated trajectory reward: 182.13±13.68 -> 292.16±11.38; 782.37±24.05 -> 786.70±26.03; 9.09±0.10 -> 19.59±0.42; 5.56±0.04 -> 17.05±0.53 | Helped |
| Use self-attention instead of LSTM (ablation) | MAPT · OpenIn., CloseOut., 3s5z, 6h vs 8z | accumulated trajectory reward: 213.42±12.74 -> 292.16±11.38; 487.12±19.64 -> 786.70±26.03; 18.43±0.32 -> 19.59±0.42; 14.54±0.33 -> 17.05±0.53 | Helped |
| Add the cooperation-aware preference layer to the LSTM variant (ablation) | MAPT* · OpenIn., CloseOut., 3s5z, 6h vs 8z | accumulated trajectory reward: 276.96±5.27 vs 213.42±12.74; 729.31±30.72 vs 487.12±19.64; 19.48±0.39 vs 18.43±0.32; 17.89±0.49 vs 14.54±0.33 | Mixed |
Surprising findings
- MAPT outperforms some multi-agent imitation learning methods trained on human demonstrations in some environments.
Limitations
- Relies on synthetic preferences from scripted teachers rather than real human labels.
- Learned reward scale differs from ground truth due to unconstrained output.
Experimental setup
Datasets: SMAC Google Research Football Bi-DexHands Multi-agent MuJoCo
Benchmarks: SMAC Google Research Football Bi-DexHands Multi-agent MuJoCo
Metrics: average accumulated trajectory reward win rate
Baselines: MAGAIL OMAR MA-MLP MA-LSTM MA-Transformer MAT
Builds on: Preference Transformer MAT Bradley-Terry preference model PPO MAPPO
Closest papers by meaning
- O-MAPL: Offline Multi-agent Preference Learning (ICML 2025 · similarity 0.94)
- Provable Reward-Agnostic Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.92)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.89)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.89)
- RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Query-Policy Misalignment in Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.