Research map / Multi-agent systems and games / Human-AI and goal-conditioned
MuMA-ToM: Multi-modal Multi-Agent Theory of Mind
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MuMA-ToM, a multi-modal multi-agent theory-of-mind video+text QA benchmark in household settings, and LIMP, a language-model-based inverse multi-agent planning method that outperforms LMMs and BIP-ALM.
Problem
Machine theory-of-mind reasoning about mental states (belief, social goal, belief of other's goal) in multi-agent, multi-modal (video + text) social interactions is not evaluated by existing benchmarks.
Approach
LIMP replaces hand-designed symbolic representations with natural language and combines GPT-4o-based likelihood estimation of agent actions/utterances under hypothesized mental variables (belief, social goal, belief of goal) with inverse multi-agent planning, after fusing VLM-extracted actions with text via an LLM.
Key results
- LIMP with GPT-4o achieves 76.6% overall accuracy on MuMA-ToM versus 56.4% for the best LMM (Gemini 1.5 Pro) and 33.9% for BIP-ALM.
- Human participants achieve 93.5% average accuracy on the benchmark.
- GPT-4o given ground-truth actions reaches only 53.2%, and LIMP without inverse planning reaches 55.3%, showing the importance of inverse planning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LIMP with GPT-4o rather than the strongest LMM baseline, Gemini 1.5 Pro | MuMA-ToM · MuMA-ToM all questions | accuracy: 56.4 -> 76.6 | Helped |
| use LIMP with GPT-4o rather than Llava 1.6 34B | MuMA-ToM · MuMA-ToM belief inference | accuracy: 93.6 -> 93.4 | No clear effect |
| use LIMP with GPT-4o rather than VideoLlama 2 7B | MuMA-ToM · MuMA-ToM social goal inference | accuracy: 45.6 -> 67.7 | Helped |
| use LIMP with GPT-4o rather than Gemini 1.5 Pro | MuMA-ToM · MuMA-ToM belief of goal inference | accuracy: 46.9 -> 68.7 | Helped |
| add inverse planning (ablation) | LIMP · MuMA-ToM all questions | accuracy: 55.3 -> 76.6 | Helped |
| provide ground-truth actions instead of using LIMP's inferred actions (ablation) | GPT-4o for ToM inference · MuMA-ToM all questions | accuracy: 76.6 -> 53.2 | Hurt |
| finetune on the training set for action captioning (ablation) | VideoLlama 2 7B · MuMA-ToM all questions | accuracy: 51.1 -> 37.3 | Hurt |
| use Llama 3.1 8B instead of GPT-4o for inverse planning (ablation) | LIMP · MuMA-ToM all questions | accuracy: 76.6 -> 33.0 | Hurt |
| add chain-of-thought prompting (ablation) | Llava 1.6 34B · MuMA-ToM all questions | accuracy: 52.8 -> 52.9 | No clear effect |
| add chain-of-thought prompting (ablation) | Llava 1.6 13B · MuMA-ToM all questions | accuracy: 43.7 -> 43.9 | No clear effect |
| add chain-of-thought prompting (ablation) | Gemini 1.5 Flash · MuMA-ToM all questions | accuracy: 42.7 -> 43.6 | No clear effect |
| add chain-of-thought prompting (ablation) | Gemini 1.5 Pro · MuMA-ToM all questions | accuracy: 56.4 -> 54.5 | Hurt |
Surprising findings
- Chain-of-thought prompting did not improve and often decreased LMM accuracy on MuMA-ToM.
- Finetuning VideoLLaMA 2 on the benchmark's training set lowered its accuracy (51.1% to 37.3%).
Limitations
- VLM hallucinations in action recognition propagate into LIMP errors and cannot be corrected by text context.
- LIMP does not explicitly model belief of belief and only supports two levels of recursive reasoning.
- Benchmark uses synthetic VirtualHome activities restricted to two agents in household settings.
Experimental setup
Datasets: MuMA-ToM training set (1,030 videos) MMToM-QA
Benchmarks: MuMA-ToM
Metrics: accuracy
Baselines: GPT-4o Gemini 1.5 Pro Gemini 1.5 Flash LLaVA 1.6 InternVL 2 VideoLlama 2 BIP-ALM SimToM
Builds on: BIP-ALM Inverse Multi-agent Planning I-POMDP GOMA chain-of-thought prompting
Closest papers by meaning
- MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems (NeurIPS 2025 · similarity 0.90)
- SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions (NeurIPS 2025 · similarity 0.89)
- AutoToM: Scaling Model-based Mental Inference via Automated Agent Modeling (NeurIPS 2025 · similarity 0.89)
- Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
- Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action (NeurIPS 2026 · similarity 0.88)
- Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.