Research map / Multi-agent systems and games / Cooperative multi-agent RL
Graph Diffusion for Robust Multi-Agent Coordination
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MCGD, a graph diffusion framework for offline multi-agent RL that combines categorical diffusion on inter-agent coordination edges with neighbor-dependent anisotropic diffusion on agent actions, evaluated on MPE, MAMuJoCo and SMAC.
Problem
Offline MARL struggles with out-of-distribution states and actions; existing diffusion-based methods diffuse each agent's trajectory independently and ignore inter-agent coordination structure, limiting robustness to dynamic changes.
Approach
Build a sparse k-nearest-neighbor coordination graph over agents, diffuse discrete edges via a similarity-derived categorical transition matrix and continuous node actions with neighbor-dependent anisotropic Gaussian noise, and sample actions with a graph denoiser guided by per-agent Q-functions.
Key results
- MCGD achieves the highest average return in all Expert/Good tasks, e.g., 3917.4 on 4ant versus 3446.5 for MADIFF, and 22.1 on SMAC 3m versus 19.6 for MADIFF.
- In shifted environments with dynamic coordination structure, MCGD improves over the best baseline by up to 14.2% (relative average gain) across MPE and MAMuJoCo settings.
- Ablations show both categorical diffusion (MCGD-CD) and anisotropic diffusion (MCGD-AD) variants outperform MADIFF on SMAC, and averaging neighbor observations beats feature concatenation in reward and compute time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use graph diffusion for multi-agent coordination instead of independent trajectory diffusion | MCGD · MPE, MAMuJoCo, and SMAC (Expert or Good datasets) | episodic return: 5.7%; 6.5%; 4.7% | Helped |
| Use graph diffusion to model coordination under dynamic agent attributes and coordination structures | MCGD · MPE and MAMuJoCo shifted environments | episodic return: 14.2% | Helped |
| Add categorical diffusion for changing coordination edges (ablation) | MCGD · SMAC 3m, 2s3z, 5m6m, and 8m | episodic return | Helped |
| Add anisotropic diffusion for neighboring-agent-dependent action noise (ablation) | MCGD · SMAC 3m, 2s3z, 5m6m, and 8m | episodic return | Helped |
| Keep a fixed k-nearest-neighbor coordination graph instead of modeling changing edge structure (ablation) | MCGD-CD · SMAC complex scenarios | episodic return | Hurt |
| Perform independent diffusion on multi-agent trajectories instead of anisotropic, coordination-dependent diffusion (ablation) | MCGD-AD · SMAC | episodic return | Hurt |
| Average neighboring observations rather than concatenate their features (ablation) | MCGD · SMAC 3m, 2s3z, 5m6m, and 8m | average reward: 22.1 ± 0.1 vs 21.3 ± 0.3; 20.7 ± 0.1 vs 20.5 ± 0.1; 18.9 ± 0.5 vs 18.4 ± 0.4; 20.1 ± 0.2 vs 19.4 ± 0.4 | Helped |
| Average neighboring observations rather than concatenate their features (ablation) | MCGD · SMAC 3m, 2s3z, 5m6m, and 8m | computation time: 174.38 ± 10.45 vs 195.40 ± 8.77; 179.01 ± 9.83 vs 203.18 ± 16.04; 177.52 ± 14.77 vs 207.54 ± 13.80; 184.74 ± 11.37 vs 21 | Helped |
Limitations
- Evaluated only on simulated benchmarks; real-world multi-robot deployment is not yet reported.
- Coordination graph is restricted to a sparse k-nearest-neighbor structure, with more complex collaboration graphs left to future work.
- Diffusion-based sampling adds inference cost relative to direct policies (timing reported only in ablation).
Experimental setup
Datasets: MPE offline datasets (Pan et al., 2022) MAMuJoCo offline datasets (Formanek et al., 2023) SMAC offline datasets (Formanek et al., 2023)
Benchmarks: MPE Spread MPE Tag MPE World MAMuJoCo 2halfcheetah MAMuJoCo 2ant MAMuJoCo 4ant SMAC 3m SMAC 2s3z SMAC 5m6m SMAC 8m
Metrics: average episodic return standard deviation over 5 seeds relative improvement
Baselines: MA-ICQ MA-CQL OMAR MA-SfBC DOM2 MADIFF
Builds on: DDPM MADIFF DOM2 categorical discrete diffusion (D3PM/DiGress) Q-loss (Wang et al.)
Closest papers by meaning
- Context-Aware Sparse Deep Coordination Graphs (ICLR 2022 · similarity 0.90)
- Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective (NeurIPS 2025 · similarity 0.90)
- Bayesian Ego-graph inference for Networked Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Bridging Training and Execution via Dynamic Directed Graph-Based Communication in Cooperative Multi-Agent Systems (AAAI 2025 · similarity 0.88)
- MADiff: Offline Multi-agent Learning with Diffusion Models (NeurIPS 2024 · similarity 0.88)
- TAPE: Leveraging Agent Topology for Cooperative Multi-Agent Policy Gradient (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.