Research map / Multi-agent systems and games / Cooperative multi-agent RL
Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIMA, a centralized multi-agent world model that treats per-timestep global state prediction as a diffusion denoising process conditioned sequentially on individual agents' actions, and uses it for learning-in-imagination MARL.
Problem
Multi-agent world models face exponentially large joint action spaces and uncertain dynamics; decentralized models misalign with the global MDP, while centralized joint models scale poorly with agent count.
Approach
Reformulate multi-agent dynamics prediction as a conditional reverse diffusion process in which each denoising step injects only one agent's action, so uncertainty about the next global state shrinks agent by agent; this yields a centralized model whose complexity scales linearly with agents, with an ELBO derivation and EDM-based training.
Key results
- DIMA attains final return of 4881 on Ant-2x4 at 1M steps vs 4471 for the next best (MARIE), and 6370 on HalfCheetah-2x3 vs 4045 for MARIE.
- On Bi-DexHands at 300K steps, DIMA reaches 259.9 on BottleCap vs 210.9 for HASAC and 367.1 on DoorOpenOutward vs 221.9 for HASAC.
- On Ant-2x4 at H=25, DIMA's observation accumulation error is 4.32 vs 6.62 for MARIE and 6.85 for MAMBA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DIMA instead of the strongest listed baseline | multi-agent policy learning · MAMuJoCo tasks at 1M steps | final episode return | Helped |
| use DIMA instead of the strongest listed baseline | multi-agent policy learning · Bi-DexHands tasks at 300K steps | final episode return | Helped |
| increase imagination horizon from H=15 to H=25 (ablation) | DIMA policy learning · HalfCheetah [6x1] | final return: 5643_{\pm 163} -> 6310_{\pm 335} | Helped |
| increase imagination horizon from H=15 to H=25 (ablation) | DIMA policy learning · Ant [4x2] | final return: 4766_{\pm 450} -> 4328_{\pm 1058} | Hurt |
| use sequential agent-wise modeling instead of joint-action conditioning (ablation) | DIMA · ShadowHandBottleCap at 100K steps | episode return: 234.1_{\pm 20.6} -> 251.8_{\pm 17.3} | Helped |
| use sequential agent-wise modeling instead of joint-action conditioning (ablation) | DIMA · ShadowHandBottleCap at 150K steps | episode return: 238.6_{\pm 22.9} -> 248.2_{\pm 11.6} | Helped |
| use sequential agent-wise modeling instead of joint-action conditioning (ablation) | DIMA · ShadowHandBottleCap at 200K steps | episode return: 246.7_{\pm 10.9} -> 246.3_{\pm 14.6} | No clear effect |
| use sequential agent-wise modeling instead of joint-action conditioning (ablation) | DIMA · ShadowHandBottleCap at 250K steps | episode return: 243.7_{\pm 18.2} -> 251.9_{\pm 12.7} | No clear effect |
| use sequential agent-wise modeling instead of joint-action conditioning (ablation) | DIMA · ShadowHandBottleCap at 300K steps | episode return: 255.2_{\pm 7.0} -> 249.2_{\pm 10.7} | No clear effect |
| use sequential agent-wise modeling instead of joint-action conditioning (ablation) | DIMA · DoorOpenOutward at 300K steps | episode return: 302.5_{\pm 76.9} -> 352.4_{\pm 40.5} | Helped |
| use sequential agent-wise modeling instead of joint-action conditioning (ablation) | DIMA · DoorOpenInward at 300K steps | episode return: 235.1_{\pm 68.1} -> 290.3_{\pm 30.4} | Helped |
| use sequential instead of joint modeling (ablation) | DIMA dynamics predictor · DoorOpenOutward, DoorOpenInward, and Pen at H=15 and H=20 | cumulative L1 observation error: 5.345_{\pm 0.267} vs 5.333_{\pm 0.273}; 7.092_{\pm 0.324} vs 7.081_{\pm 0.325}; 5.565_{\pm 0.322} vs 5.563_{\pm 0.326}; | No clear effect |
Surprising findings
- MARIE and MAMBA perform comparably to or worse than on-policy model-free MAPPO and HAPPO on several HalfCheetah settings.
- In an ablation on Bi-DexHands, sequential and joint modeling have statistically indistinguishable prediction error, and the sequential-vs-joint return gap largely vanishes at 200k-300k steps on BottleCap.
Limitations
- May face scalability challenges with hundreds of agents.
- Policy performance can degrade with longer imagination horizons (Ant 4x2 at H=25 drops to 4328 with higher variance).
- Sequential vs joint ablation shows gains mainly at low data budgets; advantage vanishes at larger budgets.
Experimental setup
Datasets: MAMuJoCo Bi-DexHands HASAC 1M-step replay dataset
Benchmarks: MAMuJoCo Bi-DexHands
Metrics: episode return cumulative L1 observation error cumulative reward error training time GPU memory
Baselines: MARIE MAMBA MAPPO HAPPO HASAC
Builds on: EDM DIAMOND MAPPO DreamerV1 FSQ
Closest papers by meaning
- Graph Diffusion for Robust Multi-Agent Coordination (ICML 2025 · similarity 0.90)
- Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL (NeurIPS 2026 · similarity 0.89)
- Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents (NeurIPS 2026 · similarity 0.89)
- Grounded Answers for Multi-agent Decision-making Problem through Generative World Model (NeurIPS 2024 · similarity 0.89)
- Beyond Prediction: Steering VLM Agents with Retrospective World Modeling (NeurIPS 2026 · similarity 0.89)
- Agent2World: Learning to Generate Symbolic World Models via Adaptive Multi-Agent Feedback (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.