Research map / Large language models / Agents, code and math
Understanding Social Reasoning in Language Models with Language Models
Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, Noah Goodman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add one-shot example to prompt (ablation) | GPT-4 · BigToM Forward Action false belief (no initial belief) | accuracy: .79 -> .98 | Improved |
| add one-shot chain-of-thought example (ablation) | GPT-4 · BigToM Backward Belief false belief (no initial belief) | accuracy: .40 -> .58 | Improved |
| add zero-shot chain-of-thought (ablation) | GPT-4 · BigToM Backward Belief false belief (no initial belief) | accuracy: .40 -> .38 | Hurt |
| add zero-shot chain-of-thought (ablation) | GPT-4 · BigToM Forward Action false belief (no initial belief) | accuracy: .79 -> .87 | Improved |
| state agent's initial belief explicitly in story (ablation) | GPT-4 · BigToM Forward Belief true belief 0-shot | accuracy: .99 -> .91 | Hurt |
| state agent's initial belief explicitly in story (ablation) | text-davinci-003 · BigToM Forward Belief TB and FB 0-shot | accuracy: .65 -> .25 | Hurt |
| use GPT-4 instead of weaker models for populating templates (ablation) | BigToM generation · template quality | generation reliability | Improved |
| add verifiers and revisers to generation pipeline with weaker models (ablation) | gpt-3.5-turbo/text-davinci-003 generation · template quality | generation quality | Mixed |
Closest papers by meaning
- Language Models Represent Beliefs of Self and Others (ICML 2024 · similarity 0.92)
- Explore Theory of Mind: program-guided adversarial data generation for theory of mind reasoning (ICLR 2025 · similarity 0.91)
- Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action (NeurIPS 2026 · similarity 0.90)
- OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling (NeurIPS 2026 · similarity 0.90)
- MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems (NeurIPS 2025 · similarity 0.90)
- Position: Theory of Mind Benchmarks are Broken for Large Language Models (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
