Research map / Multi-agent systems and games / Cooperative multi-agent RL
Understanding Individual Agent Importance in Multi-Agent System via Counterfactual Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EMAI, a MARL-based black-box explanation method that learns masking agents to randomize target agents' actions and scores each agent's per-timestep importance by the resulting reward change.
Problem
Explaining the importance of individual black-box agents at each time-step in multi-agent systems, which prior observation-level and step-level post-training explanations do not address.
Approach
Model importance via counterfactual action randomization: a CTDE-trained set of masking agents learns which target agents to randomize while minimizing the reward difference, with a sparsity reward encouraging more masking; lower masking probability indicates higher importance.
Key results
- EMAI achieves the highest RRD fidelity in all seven tasks, with relative improvements of 11% to 118% over baselines.
- Attacks guided by EMAI cause the largest reward drops, with relative improvement of 14% to 289% over baselines.
- In a user study with 36 participants, 75% said EMAI explanations better matched human intuition.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Guide attacks with EMAI-identified critical agents rather than StateMask agents | EMAI · SMAC-1c3s5z | episode team reward change: -0.25 -> -0.68 | Helped |
| Guide attacks with EMAI-identified critical agents rather than VB agents | EMAI · SMAC-27m_vs_30m | episode team reward change: -0.35 -> -1.41 | Helped |
| Guide attacks with EMAI-identified critical agents rather than VB agents | EMAI · SMAC-8m | episode team reward change: -0.53 -> -1.43 | Helped |
| Guide attacks with EMAI-identified critical agents rather than StateMask agents | EMAI · GRF-counter_attack | episode team reward change: -3.45 -> -4.45 | Helped |
| Guide attacks with EMAI-identified critical agents rather than StateMask agents | EMAI · GRF-3vs1_with_keeper | episode team reward change: -1.89 -> -2.30 | Helped |
| Guide attacks with EMAI-identified critical agents rather than GBA agents | EMAI · MPE-spread | episode team reward change: -17.54 -> -23.57 | Helped |
| Guide attacks with EMAI-identified critical agents rather than GBA agents | EMAI · MPE-reference | episode team reward change: -5.09 -> -7.18 | Helped |
| Guide policy patches with EMAI-identified critical agents rather than VB agents | EMAI · SMAC-1c3s5z | episode team reward change: +0.37 -> +0.75 | Helped |
| Guide policy patches with EMAI-identified critical agents rather than StateMask agents | EMAI · SMAC-27m_vs_30m | episode team reward change: +0.89 -> +1.11 | Helped |
| Guide policy patches with EMAI-identified critical agents rather than StateMask agents | EMAI · SMAC-8m | episode team reward change: +0.71 -> +0.92 | Helped |
| Guide policy patches with EMAI-identified critical agents rather than StateMask agents | EMAI · GRF-counter_attack | episode team reward change: +0.07 -> +1.44 | Helped |
| Guide policy patches with EMAI-identified critical agents rather than StateMask agents | EMAI · GRF-3vs1_with_keeper | episode team reward change: +0.03 -> +0.33 | Helped |
Surprising findings
- Baseline explanations in the 27m_vs_30m task have RRD close to or below 1, i.e., no better than random agent selection.
Limitations
- Practical attack and patch methods are simple and not optimized.
- Importance is based only on action impact; other abilities such as perception and planning are not captured.
Experimental setup
Datasets: SMAC Google Research Football Multi-Agent Particle Environments
Benchmarks: SMAC-1c3s5z SMAC-27m_vs_30m SMAC-8m GRF-counter_attack GRF-3vs1_with_keeper MPE-spread MPE-reference
Metrics: relative reward difference (RRD) fidelity episode team reward change under attack episode team reward change under patching
Baselines: StateMask Value-Based (VB) Gradient-Based Attribution (GBA)
Builds on: StateMask QMIX VDN QPLEX COMA
Closest papers by meaning
- Explaining Reinforcement Learning Agents through Counterfactual Action Outcomes (AAAI 2024 · similarity 0.87)
- Abstract Counterfactuals for Language Model Agents (NeurIPS 2025 · similarity 0.86)
- Counterfactual Explanations of Time Varying Rankings (Student Abstract) (AAAI 2025 · similarity 0.86)
- Generating Likely Counterfactuals Using Sum-Product Networks (ICLR 2025 · similarity 0.86)
- Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation (ICLR 2026 · similarity 0.85)
- Sequential Probabilistic Uncertainty Estimation for Parallel Multi-Agent Reasoning Systems (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.