Research map / Multi-agent systems and games / Cooperative multi-agent RL
MADiff: Offline Multi-agent Learning with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MADiff, an attention-based diffusion model for offline multi-agent learning that acts as both a decentralized policy (with teammate modeling) and a centralized controller, and also does multi-agent trajectory prediction.
Problem
Offline multi-agent learning (MARL) and multi-agent trajectory prediction require modeling inter-agent coordination; independent diffusion models lack coordination and concatenating agents' inputs is inefficient and breaks agent symmetry, while Q-learning suffers extrapolation error offline.
Approach
A U-Net diffusion model per agent with multi-head attention across agents' skip-connected latent features at each decoder layer, with shared backbone parameters; trained with a reverse diffusion loss on joint trajectories plus per-agent inverse dynamics, and sampled with classifier-free guidance and low-temperature sampling for decentralized execution with implicit teammate modeling.
Key results
- On MPE Tag Expert, MADiff-C reaches 167.6 ± 18.6 versus 116.2 ± 19.8 for OMAR, and MADiff-D reaches 120.9 ± 14.6 on the same dataset.
- On NBA trajectory prediction (length 20), MADiff-C achieves ADE 7.92 vs 15.15 for Baller2Vec++, but has higher minFDE_20 (7.61 vs 5.60).
- MADiff-D outperforms independent-diffusion ablation on MPE, with largest gap on World; teammate modeling ablation improves Spread Random from 5.7 to 19.4 normalized score.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use inter-agent attention rather than independent U-Nets (ablation) | MADiff-D · MPE tasks, especially World | average normalized score | Helped |
| share the basic U-Net parameters across agents (ablation) | MADiff-D · MPE tasks | average normalized score | No clear effect |
| add teammate modeling by training on other agents' trajectories (ablation) | MADiff-D · MPE Spread Expert | average normalized score: 93.4 ± 3.6 -> 98.4 ± 12.7 | Helped |
| add teammate modeling by training on other agents' trajectories (ablation) | MADiff-D · MPE Spread Medium | average normalized score: 35.4 ± 6.6 -> 53.2 ± 2.3 | Helped |
| add teammate modeling by training on other agents' trajectories (ablation) | MADiff-D · MPE Spread Md-Replay | average normalized score: 17.7 ± 4.3 -> 42.9 ± 11.6 | Helped |
| add teammate modeling by training on other agents' trajectories (ablation) | MADiff-D · MPE Spread Random | average normalized score: 5.7 ± 3.1 -> 19.4 ± 2.9 | Helped |
| use MADiff-C rather than Baller2Vec++ | NBA trajectory prediction · NBA trajectory length 20 | minFDE20: 5.60 ± 0.12 -> 7.61 ± 0.19 | Hurt |
| increase guidance weight among the tried values (ablation) | MADiff · offline multi-agent tasks | final performance | No clear effect |
| condition on history instead of only the current observation (ablation) | MADiff · some tasks | episodic return | Helped |
Surprising findings
- MADiff-C's minFDE_20 on NBA is worse than Baller2Vec++ (7.61 vs 5.60), despite better one-shot metrics.
- On SMACv2, MADiff-D underperforms MAICQ only when both position and unit-type stochasticity are present.
Limitations
- Decentralized version requires each agent to predict all teammates' trajectories, which is unsuitable for tens or hundreds of agents.
- Sequence-modeling nature makes it susceptible to high-stochasticity environments, where it underperforms Q-learning methods (e.g., SMACv2 with both randomness types).
- Experiments limited to at most 8 agents.
Experimental setup
Datasets: MPE offline datasets (OMAR) MA Mujoco offline datasets (off-the-grid) SMAC offline datasets (off-the-grid) NBA player tracking dataset (2015-16 season) SMACv2 terran_5_vs_5 datasets
Benchmarks: MPE (Spread, Tag, World) MA Mujoco (2halfcheetah, 2ant, 4ant) SMAC (3m, 2s3z, 5m6m, 8m) NBA trajectory prediction SMACv2 terran_5_vs_5
Metrics: normalized average episodic return ADE FDE minADE_20 minFDE_20
Baselines: BC MA-ICQ MA-TD3+BC MA-CQL OMAR MADT Baller2Vec++ MAICQ
Builds on: Diffuser Decision Diffuser Multi-Agent Decision Transformer (MADT) classifier-free guidance OMAR
Closest papers by meaning
- ATraDiff: Accelerating Online Reinforcement Learning with Imaginary Trajectories (ICML 2024 · similarity 0.88)
- Graph Diffusion for Robust Multi-Agent Coordination (ICML 2025 · similarity 0.88)
- Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization (AAAI 2025 · similarity 0.87)
- Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective (NeurIPS 2025 · similarity 0.87)
- ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization (ICLR 2025 · similarity 0.86)
- Reasoning with Latent Diffusion in Offline Reinforcement Learning (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.