Research map / Vision-language and multimodal / Robot action and navigation
SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SoMi-ToM, a multi-perspective embodied theory-of-mind benchmark built from three LVLM agents crafting in Minecraft, and finds LVLMs trail humans by 40.1% (first-person) and 26.4% (third-person).
Problem
Most theory-of-mind benchmarks are static and text-based or single-perspective, lacking embodied, multi-agent social interactions with collaboration and covert obstruction; existing multimodal ToM tests do not measure real-time state inference or multi-perspective inference.
Approach
Generate multi-agent Minecraft interaction logs with LVLM agents (collaborative and covert-obstruction relationships) and build multiple-choice ToM questions from both a first-person view (real-time state inference from screenshots and dialogue) and a third-person view (goal and behavior inference from full videos).
Key results
- In first-person evaluation, the best LVLM (GPT-4o with CoT) reached 59.5% weighted accuracy versus 90.0% for humans, a 40.1-point gap.
- In third-person evaluation, GPT-4o with CoT reached 82.9% average accuracy versus 93.3% for humans, a gap of about 26.4 points.
- CoT improved most models in first-person state inference but decreased accuracy for most models in third-person evaluation, except Gemini 2.0 Flash, GPT-4o and Qwen2.5-VL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Chain-of-Thought prompting (ablation) | LVLMs on SoMi-ToM first-person state inference · SoMi-ToM first-person state inference | weighted average accuracy: 40.6 ± 5.7 -> 49.9 ± 9.2 | Helped |
| Add Chain-of-Thought prompting (ablation) | GPT-4o · SoMi-ToM first-person state inference | weighted average accuracy: 35.6 -> 59.5 | Helped |
| Add Chain-of-Thought prompting (ablation) | InternVL2.5 78B · SoMi-ToM first-person state inference | weighted average accuracy: 45.7 -> 44.6 | Hurt |
| Add Chain-of-Thought prompting (ablation) | GPT-4o · SoMi-ToM third-person ToM | average accuracy: 69.7 ± 9.6 -> 82.9 ± 6.4 | Helped |
| Add Chain-of-Thought prompting (ablation) | Gemini 2.0 Flash · SoMi-ToM third-person ToM | average accuracy: 78.3 ± 13.8 -> 79.4 ± 13.0 | Helped |
| Add Chain-of-Thought prompting (ablation) | Qwen2.5-VL · SoMi-ToM third-person ToM | average accuracy: 78.3 ± 16.7 -> 80.6 ± 12.4 | Helped |
| Add Chain-of-Thought prompting (ablation) | Gemini 1.5 Pro · SoMi-ToM third-person ToM | average accuracy: 75.4 ± 13.5 -> 71.4 ± 16.5 | Hurt |
| Add Chain-of-Thought prompting (ablation) | InternVL2.5 78B · SoMi-ToM third-person ToM | average accuracy: 76.0 ± 7.7 -> 72.0 ± 12.0 | Hurt |
| Add Chain-of-Thought prompting (ablation) | VideoLLaMA 3 7B · SoMi-ToM third-person ToM | average accuracy: 37.7 ± 10.4 -> 36.0 ± 13.5 | Hurt |
| Add Chain-of-Thought prompting (ablation) | LLaVA-Video 7B · SoMi-ToM third-person ToM | average accuracy: 50.3 ± 17.7 -> 48.6 ± 16.6 | Hurt |
Surprising findings
- Chain-of-thought lowered third-person accuracy for most evaluated models, with only GPT-4o, Gemini 2.0 Flash and Qwen2.5-VL improving.
- Self-ToM state inference was easier for LVLMs than inferring others' states.
Limitations
- Environment and social dynamics are simplified: only unidirectional obstruction and 5 crafting goals with three agents.
- Third-person videos are human-recorded with selective camera focus, and the dataset is small (35 videos, 175 third-person questions).
- Models evaluated are mostly API-based with varying input settings (frame sampling, images vs video), which complicates comparison.
Experimental setup
Datasets: SoMi-ToM (Minecraft multi-agent interaction logs)
Benchmarks: SoMi-ToM first-person state inference SoMi-ToM third-person goal and behavior inference
Metrics: accuracy
Baselines: Human (21 participants) Gemini 1.5 Pro Gemini 2.0 Flash GPT-4o InternVL2.5 78B Qwen2.5-VL LLaVA 1.6 13B VideoLLaMA 3 7B LLaVA-Video 7B
Builds on: Mindcraft Chain-of-Thought prompting MMToM-QA Phase
Closest papers by meaning
- MuMA-ToM: Multi-modal Multi-Agent Theory of Mind (AAAI 2025 · similarity 0.89)
- Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
- HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs (AAAI 2026 · similarity 0.87)
- MimeQA: Towards Socially-Intelligent Nonverbal Foundation Models (NeurIPS 2025 · similarity 0.87)
- Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner (ICML 2025 · similarity 0.87)
- EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.