Research map / Multi-agent systems and games / Cooperative multi-agent RL
R3DM: Enabling Role Discovery and Diversity Through Dynamics Models in Multi-agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes R3DM, a role-based MARL framework that derives roles from past trajectories via contrastive learning and adds intrinsic rewards from a role-conditioned DreamerV3-style dynamics model to diversify future behavior across roles.
Problem
Existing role-based MARL methods derive roles only from agents' past observation-action histories, ignoring how roles should shape future behavior, which leads to redundant behavior and weak coordination.
Approach
Maximize a mutual-information objective linking roles, past trajectories and expected future trajectories; obtain roles via contrastive learning (ACORM-style) on histories and add policy-diversity and role-conditioned dynamics intrinsic rewards computed from RSSM world models.
Key results
- Improves win rates by up to 20% on SMAC and SMACv2 hard maps such as 3s5z_vs_3s6z and protoss_10_vs_11 relative to baselines.
- Outperforms ACORM, its base method, on many environments, and outperforms CDS and EMC on SMACv2.
- Ablation on 3s5z_vs_3s6z: shorter imagination horizons (k=1,2) outperform longer ones (k=5,10); 3 roles gives faster convergence; removing contrastive learning reduces performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use role-conditioned dynamics-model intrinsic rewards and contrastive role learning | ACORM-based MARL · SMAC and SMACv2 | win rate: up to 20% | Helped |
| Use R3DM instead of ACORM | Role-based MARL · SMAC | test win rate | Helped |
| Use R3DM instead of CDS | MARL · SMACv2 | test cumulative reward | Helped |
| Use R3DM instead of EMC | MARL · SMACv2 asymmetric 10_vs_11 environments | learning progress | Helped |
| Use R3DM instead of ACORM | Role-based MARL · SMACv2 5_vs_5 suite | test win rate | No clear effect |
| Use R3DM instead of ACORM | Role-based MARL · SMACv2 5_vs_5 suite | test cumulative reward | Helped |
| Use a one-step imagination horizon instead of longer horizons (ablation) | R3DM · 3s5z_vs_3s6z | performance | Helped |
| Increase imagination horizon from k=1 to k=2 (ablation) | R3DM · 3s5z_vs_3s6z | performance | No clear effect |
| Increase imagination horizon to 5 or 10 steps (ablation) | R3DM · 3s5z_vs_3s6z | performance | Hurt |
| Use 3 role clusters instead of higher role cardinalities (ablation) | R3DM · 3s5z_vs_3s6z | sample efficiency | Helped |
| Vary role cardinality from 2 to 8 (ablation) | R3DM · 3s5z_vs_3s6z | final performance | No clear effect |
| Remove contrastive learning while retaining intrinsic rewards (ablation) | R3DM · 3s5z_vs_3s6z | performance | Hurt |
Surprising findings
- Longer imagination horizons degrade performance, attributed to compounding dynamics-model errors.
- Final performance is statistically indistinguishable across role cardinalities 2 to 8, with only sample efficiency differing.
Limitations
- Number of roles must be fixed a priori as a hyperparameter.
- Dynamics model uses only local observation-action history and ignores influence of other agents' actions and roles.
- Longer imagination horizons cause compounding prediction errors and degraded performance.
Experimental setup
Datasets: SMAC SMACv2
Benchmarks: SMAC SMACv2
Metrics: test win rate test cumulative reward
Baselines: QMIX ACORM CIA GoMARL CDS EMC
Builds on: ACORM QMIX DreamerV3 MoCo ROMA
Closest papers by meaning
- Attention-Guided Contrastive Role Representations for Multi-agent Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Foundations of Multi-Agent Learning in Dynamic Environments: Where Reinforcement Learning Meets Strategic Decision-Making (AAAI 2025 · similarity 0.88)
- HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.88)
- Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.87)
- Dynamic Optimizations of LLM Ensembles with Two-Stage Reinforcement Learning Agents (ICML 2026 · similarity 0.87)
- Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.