Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Risk-Sensitive Theory of Mind: Coordinating with Agents of Unknown Bias using Cumulative Prospect Theory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RS-ToM, a zero-shot framework where an agent trains CPT-based risk-sensitive candidate policies with a deep MARSRL (DDQN + QRE) and infers its partner's risk-sensitivity via Bayesian belief updates, evaluated in a new Risky Overcooked benchmark.
Problem
Autonomous agents usually model human partners as noisy-rational, which fails when partners make risk-sensitive biased decisions in risky coordination tasks; prior CPT-based personalization needs costly human data priors or is limited to tabular methods.
Approach
Train a space of deep multi-agent policies under different CPT parameterizations (risk-seeking, risk-neutral, risk-averse) using a CPT-value Bellman update with a novel replay memory storing all possible next-state prospects, then have the ego agent perform Bayesian inference over these candidate policies to align with an unknown partner online without data priors.
Key results
- RS-ToM improved team reward over the noisy-rational baseline by +77% in RCR and +189% in RMP on average across partners, performing nearly identically to the Oracle.
- RS-ToM improved coordination fluency (C-ACT) by 104% on average and more than doubled partner predictability (+121%) relative to RAT.
- With RAT, risk-taking moved in the wrong direction relative to the partner (Averse partner saw increased risks, Seeking saw decreased), while RS-ToM matched Oracle risk-taking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RS-ToM instead of RAT to adapt to the partner’s unknown risk-sensitivity | Risky Overcooked coordination · RCR | team reward Σr: +77% | Helped |
| Use RS-ToM instead of RAT to adapt to the partner’s unknown risk-sensitivity | Risky Overcooked coordination · RMP | team reward Σr: +189% | Helped |
| Use RS-ToM instead of RAT to adapt to the partner’s unknown risk-sensitivity | Risky Overcooked coordination · RCR and RMP | C-ACT: +104% | Helped |
| Infer the partner’s risk-sensitivity and align the team’s risk-taking | Risky Overcooked coordination · RCR and RMP | risk-taking alignment | Helped |
| Use RS-ToM instead of RAT to predict a risk-sensitive partner | Risky Overcooked coordination · RCR and RMP | partner predictability PH: +121% | Helped |
Surprising findings
- Noisy-rational agents increased risk-taking when partner was risk-averse and decreased it when partner was risk-seeking, the opposite of the partner's preferences.
Limitations
- Evaluation only in simulation with an artificial partner; no real human study.
- Only three discrete candidate policies and two layouts; generalization to continuous risk-sensitivity and other layouts untested.
- Tabular-like discrete strategy set means alternative explanations (e.g., limited planning horizon) are not separated from risk-sensitivity.
Experimental setup
Datasets: Risky Overcooked (self-generated simulated environment)
Benchmarks: Risky Overcooked (RCR and RMP layouts)
Metrics: cumulative team reward concurrent activity (C-ACT) number of risks taken partner predictability
Baselines: Noisy-rational (RAT) assumption Oracle (known partner risk-sensitivity)
Builds on: Cumulative Prospect Theory Double Deep Q-Network Overcooked Nash Q-Learning with CPT (Danis et al., 2023)
Closest papers by meaning
- Emergent Risk Awareness in Rational Agents under Resource Constraints (NeurIPS 2025 · similarity 0.87)
- Adaptive Theory of Mind for LLM-based Multi-Agent Coordination (AAAI 2026 · similarity 0.87)
- Training Generalizable Collaborative Agents via Strategic Risk Aversion (NeurIPS 2026 · similarity 0.86)
- Mental-Models for Multi-Agent Systems (NeurIPS 2026 · similarity 0.86)
- Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action (NeurIPS 2026 · similarity 0.85)
- Undermining Mental Proof: How AI Can Make Cooperation Harder by Making Thinking Easier (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.