Research map / Multi-agent systems and games / Cooperative multi-agent RL
Intrinsic Action Tendency Consistency for Cooperative Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IAM, an action-model-based intrinsic reward that makes agents match neighbors' predictions of the central agent's action tendency, and a reward-additive CTDE (RA-CTDE) formulation to inject N per-agent intrinsic rewards into QMIX and VDN.
Problem
Cooperative MARL under CTDE suffers from low sample efficiency because agents' action tendencies diverge, and sparse team rewards give no explicit consensus signal; existing intrinsic rewards are mostly exploration-oriented or are averaged into one global reward.
Approach
Each agent has action models that predict a central agent's action tendency (Q-values); the L2 discrepancy between predictions and the central Q-function gives a cooperative intrinsic reward. RA-CTDE factorizes the global TD target into N per-agent targets (proved gradient-equivalent to CTDE) so each agent receives its own intrinsic reward term.
Key results
- QMIX-IAM outperforms baselines including QPLEX, Qatten and EMC-style rewards in most SMAC scenarios, especially very hard maps requiring team coordination.
- IAM improves QMIX and VDN on sparse-reward GRF academy scenarios (run_pass_and_shoot_with_keeper and pass_and_shoot_with_keeper).
- Empirically, RA-CTDE without intrinsic rewards matches CTDE performance (QMIX-RA, VDN-RA) on 2s3z, 8m_vs_9m and MMM2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the action-model intrinsic reward using RA-CTDE | QMIX · SMAC | performance | Helped |
| add the action-model intrinsic reward using RA-CTDE | QMIX · GRF Academy_run_pass_and_shoot_with_keeper and Academy_pass_and_shoot_with_keeper | performance | Helped |
| add the action-model intrinsic reward using RA-CTDE | VDN · GRF Academy_run_pass_and_shoot_with_keeper and Academy_pass_and_shoot_with_keeper | performance | Helped |
| add the action-model intrinsic reward | VDN · SMAC | performance | Helped |
| use the action-model intrinsic reward instead of the world-model intrinsic reward | SACD · MPE Cooperative Navigation | test occupancy | Helped |
| replace the action-model reward with the exploration-based EMC reward (ablation) | QMIX · SMAC 6h_vs_8z and 3s_vs_5z | performance | Helped |
| replace the action-model reward with the exploration-based EMC reward (ablation) | QMIX · SMAC maps other than 6h_vs_8z and 3s_vs_5z | performance | No clear effect |
| replace the action-model reward with the world-model intrinsic reward (ablation) | QMIX · SMAC 3s5z | performance | Helped |
| replace the action-model reward with the world-model intrinsic reward (ablation) | QMIX · SMAC maps other than 3s5z | performance | No clear effect |
| replace per-agent RA-CTDE reward addition with centralized reward addition (ablation) | VDN-IAM and QMIX-IAM · SMAC 8m_vs_9m and 3s5z_vs_3s6z | performance | Hurt |
| use RA-CTDE without additional intrinsic rewards instead of standard CTDE (ablation) | QMIX and VDN · SMAC 2s3z, 8m_vs_9m, and MMM2 | performance | No clear effect |
| add the action-model reward rather than an exploration-based intrinsic reward (ablation) | QMIX and VDN with RA-CTDE · SMAC 8m_vs_9m | performance | Helped |
Surprising findings
- Exploration-based intrinsic rewards (curiosity, RND) did not generalize well in cooperative SMAC tasks compared with the action-model reward.
- Averaging intrinsic rewards into a single shared global reward (EMC-style) underperforms the per-agent reward-additive integration.
Limitations
- Only discrete state-action environments were considered; continuous spaces not addressed.
- No specific design for heterogeneous agents.
- Action model relies on imagined observations via viewpoint switching that depends on observation structure (SMAC, GRF, MPE).
Experimental setup
Datasets: SMAC Google Research Football Multi-Agent Particle Environment
Benchmarks: SMAC Google Research Football Multi-Agent Particle Environment
Metrics: win rate test occupancy rate
Baselines: QPLEX Qatten QMIX VDN EMC LIIR world model intrinsic reward (ELIGN) RND
Builds on: QMIX VDN IQL EMC
Closest papers by meaning
- GTDE: Grouped Training with Decentralized Execution for Multi-agent Actor-Critic (AAAI 2025 · similarity 0.89)
- Stateful Active Facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learning (ICLR 2023 · similarity 0.88)
- Distributionally Robust Cooperative Multi-agent Reinforcement Learning with Value Factorization (ICLR 2026 · similarity 0.88)
- Locality Matters: A Scalable Value Decomposition Approach for Cooperative Multi-Agent Reinforcement Learning (AAAI 2022 · similarity 0.88)
- Multi-agent cooperation through learning-aware policy gradients (ICLR 2025 · similarity 0.87)
- AgentMixer: Multi-Agent Correlated Policy Factorization (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.