Research map / Large language models / Agents, code and math

OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling

NeurIPS 2026 · Large language models › Agents, code and math · approach: Mathematical and structural reasoning benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OmniToM, a ToM benchmark of 895 ToMBench-derived stories with 22,343 labeled belief propositions, evaluating LLMs on explicit belief extraction (Stage 1) and seven-dimension belief labeling (Stage 2) rather than endpoint QA.

Problem

Endpoint question answering for Theory of Mind in LLMs does not reveal whether models construct the underlying actor-specific mental-state representations needed for robust social reasoning, especially with false, evolving, or nested beliefs.

Approach

Represent each story as actor-indexed belief propositions and evaluate models on extracting them (bidirectional MatchCount F1 with an LLM semantic judge) and labeling each with a seven-dimensional ATOMS-grounded schema (order, truth status, knowledge access, representation, content type, mental source, context), with annotation done by a human-calibrated LLM pipeline.

Key results

• Stage 2 belief-labeling accuracy reaches 85.95% for Gemini-2.5 Flash, with GPT-5 at 82.85%.
• Stage 1 belief-extraction macro F1 peaks at 57.69% for Gemma-3 27B.
• Knowledge Access (56.13-74.59%) and Representation (54.33-87.58%) are the weakest Stage 2 dimensions across models.

Surprising findings

• Open-weight Gemma-3 27B achieves the top Stage 1 F1 while Gemini-2.5 Flash leads Stage 2 labeling, and GPT-5 is excluded from Stage 1 because it serves as the judge.
• Order-1 actor beliefs, not higher-order ones, are the hardest to label (71.2% overall), with Order 2+ recovering to 74.4%.

Limitations

• Text-only, short self-contained stories from seven ToMBench categories; no multimodal or long-horizon tracking; belief order capped at 3.
• Stage 1 semantic judge (GPT-5) has only moderate human agreement (72.03%), so F1 is approximate.
• Schema labels retain some interpretive subjectivity despite human verification.

Experimental setup

Datasets: ToMBench

Benchmarks: OmniToM

Metrics: F1 (MatchCount-based) exact-match labeling accuracy per-dimension accuracy

Baselines: Gemini-2.5 Flash GPT-5 Gemma-3 27B Llama-3.1 8B Llama-3.3 70B Mistral-Small 24B Mistral-Large 123B Qwen3 8B Qwen3 32B

Builds on: ToMBench ATOMS TELeR SymbolicToM

Closest papers by meaning

• ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind (AAAI 2025 · similarity 0.91)
• Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action (NeurIPS 2026 · similarity 0.90)
• Explore Theory of Mind: program-guided adversarial data generation for theory of mind reasoning (ICLR 2025 · similarity 0.89)
• Reflection-Bench: Evaluating Epistemic Agency in Large Language Models (ICML 2025 · similarity 0.89)
• Language Models Represent Beliefs of Self and Others (ICML 2024 · similarity 0.89)
• AutoToM: Scaling Model-based Mental Inference via Automated Agent Modeling (NeurIPS 2025 · similarity 0.88)