Research map / Multi-agent systems and games / Games and equilibria
Explaining Decisions of Agents in Mixed-Motive Games
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes three explanation methods (SBUE, probable-actions, SICA) for agents' decisions in mixed-motive games and validates them with human user studies in no-press Diplomacy and an LLM-based prisoner's-dilemma-style game (COP).
Problem
Explaining agents' decisions in mixed-motive multi-agent games, where cooperation and competition interleave, and communication occurs via implicit actions or cheap talk; existing cooperative-setting explanation methods do not address competition, cheap talk, or implicit communication.
Approach
Three simulation-based explanations: Strategy-Based Utility Explanations estimate expected utility of an action for all agents; probable-actions explanations show likely responses of other agents; Shared Interests Correlation Analysis computes Pearson correlations of agents' simulated utilities to indicate friendliness or hostility.
Key results
- SICA achieved MAP@K of 0.74 (enemies) and 0.66 (friends) for expert Diplomacy annotations, versus 0.55-0.63 for random and centers baselines.
- In the Diplomacy user study, the combined explanation (C) raised willingness to choose the superior strategy in 65% of cases and improved understanding of friends/enemies in 41% of cases.
- In the COP user study, explanations shifted users' message choice in 73% of cases and improved understanding of inter-agent relationships in 59% of cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SICA rankings instead of random rankings | SICA agent-relationship explanations · Diplomacy expert annotations, enemies | MAP@K: 0.55 -> 0.74 | Helped |
| Use SICA rankings instead of the centers heuristic | SICA agent-relationship explanations · Diplomacy expert annotations, enemies | MAP@K: 0.63 -> 0.74 | Helped |
| Use SICA rankings instead of random rankings | SICA agent-relationship explanations · Diplomacy non-expert annotations, enemies | MAP@K: 0.56 -> 0.81 | Helped |
| Use SICA rankings instead of random rankings | SICA agent-relationship explanations · Risk enemies | MAP@K: 0.61 -> 0.81 | Helped |
| Show OS explanations | Diplomacy users choosing a higher-utility strategy · Diplomacy H1 (+) | Proportion higher: 0.57 | Helped |
| Show SU explanations | Diplomacy users choosing a higher-utility strategy · Diplomacy H1 (+) | Proportion higher: 0.65 | Helped |
| Combine OS and SU explanations | Diplomacy users' understanding of interests and strategy effects · Diplomacy H2 (+) | Proportion higher: 0.23 -> 0.41 | Helped |
| Show OS, SU, and combined explanations | Users selecting the inferior strategy less often · Diplomacy H1 (-) | Proportion lower: OS 0.52; SU 0.43; C 0.57 | Helped |
| Show SICA and SBUE explanations | COP users choosing which message to send · COP H3 (+) | Proportion higher: 0.73 | Helped |
| Show SICA and SBUE explanations | COP users' understanding of inter-agent relationships · COP H4 (+) | Proportion higher: 0.59 | Helped |
| Show SICA and SBUE explanations | COP users' understanding of message influence · COP H5 (+) | Proportion higher: 0.30 | Helped |
| Add SICA and SBUE explanation information to the prompt | GPT-4 COP agents choosing between messages · COP message choice | Rate choosing higher-utility message m1: 47% -> 91% | Helped |
Surprising findings
- Greedy decoding (temperature 0) for probable-actions explanations of LLM agents sometimes produced outcomes that were not the most probable at temperature 0.7, so the explanations can be misleading.
- In the LLM study, GPT-4 agents' first-message bias was reduced when explanations were provided, but an explanation-free bias toward the first message remained when agents were indifferent.
Limitations
- Subject is understudied, making comparison with established baselines difficult.
- Probable-actions explanations with greedy decoding for LLM agents can be misleading.
- User study participants were predominantly male, limiting generalizability.
Experimental setup
Datasets: No-press Diplomacy game environment Risk (simplified 4-player version) Communicate Out of Prison (COP) simulated games
Benchmarks: No-press Diplomacy human annotations Risk human annotations COP user study
Metrics: MAP@K RMSE cosine similarity Wilcoxon signed-rank p-value proportion of raised/lowered Likert answers
Baselines: random ranking centers-based heuristic ranking
Builds on: SAVE (Simulation Action Value Estimation) Shapley values interdependence theory
Closest papers by meaning
- Explaining Reinforcement Learning Agents through Counterfactual Action Outcomes (AAAI 2024 · similarity 0.86)
- Aligning Individual and Collective Objectives in Multi-Agent Cooperation (NeurIPS 2024 · similarity 0.86)
- Learning to Balance Altruism and Self-interest Based on Empathy in Mixed-Motive Games (NeurIPS 2024 · similarity 0.85)
- Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation (NeurIPS 2024 · similarity 0.85)
- Narrowing the Collaboration Gap, Probably (NeurIPS 2026 · similarity 0.85)
- Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordia (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.