Research map / Large language models / AI and society
Position: Theory of Mind Benchmarks are Broken for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Argues that LLM theory of mind benchmarks are broken because they measure literal (prediction) rather than functional (adaptive decision-making) theory of mind, and supports this with repeated-game experiments against simple partner policies.
Problem
Existing LLM theory of mind benchmarks test only whether models predict other agents' behavior (literal theory of mind), not whether they adapt their own actions to partners in context (functional theory of mind).
Approach
Defines literal theory of mind (prediction accuracy) and functional theory of mind (regret of the agent's own policy relative to the optimal response to a partner), and shows with repeated matrix games that LLMs can predict partner actions well yet respond poorly, while a tabular RMax baseline adapts well.
Key results
- On Rock-Paper-Scissors against single-action partners, LLaMA-2 70B reaches ToM accuracy of 96.8% yet has functional regret per step of 0.971, versus 0.083 for the tabular RMax model.
- Mixtral 8x7B CoT prompting on RPS gives functional regret 0.648 while its rational-policy regret from predictions is 0.998 with ToM accuracy only 33.5%.
- DeepSeek-R1-Distill-Qwen-32B achieves functional regret 0.074 on RPS single-action partners, competitive with tabular 0.083, while its ToM accuracy is only 63.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Tabular RMax instead of Mixtral 8x7B Instruct v1 | RPS against single-action partners · RPS, single-action partners | functional regret per step: 0.542 ± 0.070 -> 0.083 ± 0.004 | Helped |
| use LLaMA-2 Chat rather than its non-Chat counterpart | LLaMA-2 models on RPS · RPS, single-action partners | functional regret per step: 0.971 ± 0.067 -> 0.857 ± 0.142; 1.015 ± 0.044 -> 0.891 ± 0.173; 0.972 ± 0.039 -> 0.904 ± 0.151 | Mixed |
| add chain-of-thought prompting instead of QA prompting (ablation) | LLaMA-3 70B Instruct · RPS, IBS, and IPD, single-action partners | functional regret per step: 0.444 ± 0.107 -> 0.213 ± 0.015; 1.391 ± 0.283 -> 2.475 ± 0.208; 0.996 ± 0.133 -> 0.892 ± 0.210 | Mixed |
| add three ideal reasoning examples to CoT prompting (ablation) | LLaMA-3 70B Instruct · RPS, IBS, and IPD, single-action partners | functional regret per step: 0.213 ± 0.015 -> 0.121 ± 0.017; 2.475 ± 0.208 -> 0.526 ± 0.081; 0.892 ± 0.210 -> 2.773 ± 0.730 | Mixed |
| add Social QA partner-action prediction (ablation) | LLaMA-3 70B Instruct · RPS, IBS, and IPD, single-action partners | functional regret per step: 0.444 ± 0.107 -> 0.256 ± 0.017; 1.391 ± 0.283 -> 1.613 ± 0.163; 0.996 ± 0.133 -> 0.550 ± 0.071 | Mixed |
| give the model the partner's actual action instead of a prediction (ablation) | LLaMA-3 70B Instruct · RPS, IBS, and IPD, single-action partners | functional regret per step: 0.444 ± 0.107 -> 0.238 ± 0.046; 1.391 ± 0.283 -> 1.343 ± 0.146; 0.996 ± 0.133 -> 0.839 ± 0.056 | Mixed |
| emphasize maximizing reward given the oracle partner action (ablation) | LLaMA-3 70B Instruct with Oracle prompting · RPS, IBS, and IPD, single-action partners | functional regret per step: 0.238 ± 0.046 -> 0.153 ± 0.014; 1.343 ± 0.146 -> 0.785 ± 0.072; 0.839 ± 0.056 -> 0.767 ± 0.100 | Helped |
| remove interaction history from Oracle + Max (ablation) | LLaMA-3 70B Instruct · RPS, IBS, and IPD, single-action partners | functional regret per step: 0.153 ± 0.014 -> 1.275 ± 0.152; 0.785 ± 0.072 -> 2.060 ± 0.079; 0.767 ± 0.100 -> 0.206 ± 0.018 | Mixed |
| remove payoff information from Oracle + Max (ablation) | LLaMA-3 70B Instruct · RPS, IBS, and IPD, single-action partners | functional regret per step: 0.153 ± 0.014 -> 0.103 ± 0.015; 0.785 ± 0.072 -> 0.883 ± 0.097; 0.767 ± 0.100 -> 0.241 ± 0.046 | Mixed |
| add Social QA partner-action prediction (ablation) | Mixtral 8x7B Instruct v1 · RPS, IBS, and IPD, single-action partners | functional regret per step: 0.542 ± 0.070 -> 0.676 ± 0.058; 2.055 ± 0.391 -> 2.557 ± 0.451; 0.949 ± 0.142 -> 1.569 ± 0.132 | Mixed |
| use DeepSeek-R1-Distill-Qwen-32B instead of Tabular RMax | single-action partners · RPS, IBS, and IPD | functional regret per step: 0.083 ± 0.004 -> 0.074 ± 0.009; 0.211 ± 0.012 -> 0.126 ± 0.045; 0.086 ± 0.009 -> 0.121 ± 0.014 | Mixed |
| use DeepSeek-R1-Distill-Qwen-32B instead of Tabular RMax | tit-for-tat partners · RPS, IBS, and IPD | functional regret per step: 0.224 ± 0.007 -> 0.906 ± 0.041; 0.468 ± 0.031 -> 0.045 ± 0.011; 0.248 ± 0.005 -> 4.789 ± 0.061 | Mixed |
Surprising findings
- LLMs predict partner actions with high accuracy yet choose near-uniform (Nash-like) actions against an always-Rock opponent.
- Giving LLaMA-3 70B the oracle partner action and payoffs still underperforms the tabular RMax model learned from scratch.
Limitations
- Evaluation limited to simple matrix games with simple partner policies, not complex social benchmarks.
- Position paper; claims that existing benchmarks are 'broken' are argued rather than exhaustively tested across existing benchmarks.
- Some experiments use only one or a few models, and the tabular baseline is not an LLM.
Experimental setup
Benchmarks: Rock-Paper-Scissors Iterated Battle of Sexes Iterated Prisoner's Dilemma
Metrics: functional regret per step theory of mind prediction accuracy (ToM %)
Baselines: Tabular RMax Chain-of-thought prompting Reflexion Plans + Insights Social prompting System 2 Attention
Builds on: Reflexion Chain-of-thought prompting System 2 Attention RMax
Closest papers by meaning
- Reflection-Bench: Evaluating Epistemic Agency in Large Language Models (ICML 2025 · similarity 0.89)
- Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action (NeurIPS 2026 · similarity 0.89)
- OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling (NeurIPS 2026 · similarity 0.88)
- ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind (AAAI 2025 · similarity 0.88)
- Language Models Represent Beliefs of Self and Others (ICML 2024 · similarity 0.88)
- Explore Theory of Mind: program-guided adversarial data generation for theory of mind reasoning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.