Research map / Multi-agent systems and games / LLM agents
AutoToM: Scaling Model-based Mental Inference via Automated Agent Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoToM, which automatically discovers agent models (Bayesian networks over mental variables and timesteps) and performs LLM-backed Bayesian inverse planning for theory-of-mind inference, outperforming LLMs and large reasoning models on five ToM benchmarks.
Problem
Theory of Mind inference with LLMs is prone to systematic errors, while model-based Bayesian inverse planning methods need handcrafted, domain-specific agent models and fail to generalize across domains.
Approach
An LLM proposes an initial agent model, performs Bayesian inverse planning with LLM-estimated local conditionals, and iteratively adds mental variables (goal, belief, observation, interactive state) or earlier timesteps guided by a utility that trades off posterior entropy against model complexity.
Key results
- AutoToM (GPT-4o backend) reaches 82.43% average accuracy across five ToM benchmarks, versus 63.39% for GPT-4o and 74.18% for Gemini 2.0 Flash Thinking.
- On MMToM-QA, AutoToM improves over GPT-4o from 44.0% to 83.0%, and with Qwen3-235B, DeepSeek-V3 and Gemini-2.5-Flash backends it gains 22-37 points over the raw LLM.
- In online goal inference for embodied assistance, AutoToM achieves a 27.7% speedup versus 6.8% for GPT-4o and 6.3% for random goals.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AutoToM instead of SymbolicToM | ToMi · ToMi | accuracy: 98.60 -> 88.30 | Hurt |
| Use AutoToM instead of o3-mini-high | BigToM · BigToM | accuracy: 86.92 -> 86.92 | No clear effect |
| Use AutoToM instead of o3-mini-high | MMToM-QA · MMToM-QA | accuracy: 64.67 -> 83.00 | Helped |
| Use AutoToM instead of Gemini 2.0 Flash Thinking | MuMA-ToM · MuMA-ToM | accuracy: 82.56 -> 81.44 | Hurt |
| Use AutoToM instead of o3-mini-high | Hi-ToM · Hi-ToM | accuracy: 75.00 -> 72.50 | Hurt |
| Use AutoToM with GPT-4o backend instead of GPT-4o alone | FANToM · FANToM (200 false-belief first-order questions) | accuracy: 57.5% -> 72.7% | Helped |
| Use AutoToM with GPT-4o backend instead of GPT-4o alone | OpenToM · OpenToM attitude questions | Macro-F1: 0.48 -> 0.56 | Helped |
| Use AutoToM online goal inference instead of GPT-4o inference | O-WAH helper planner · Online Watch-And-Help (O-WAH) | speedup: 6.8% -> 27.7% | Helped |
| Add hypothesis reduction (ablation) | AutoToM · ToM benchmarks (all) | accuracy: 81.15 -> 82.43 | Helped |
| Use adaptive model selection instead of always using POMDP (ablation) | AutoToM · ToM benchmarks (all) | accuracy: 72.59 -> 82.43 | Helped |
| Add variable adjustment (ablation) | AutoToM · ToM benchmarks (all) | accuracy: 77.49 -> 82.43 | Helped |
| Add timestep adjustment instead of considering only the last timestep (ablation) | AutoToM · ToM benchmarks (all) | accuracy: 69.11 -> 82.43 | Helped |
Surprising findings
- BIP-ALM and LIMP, the prior LLM+BIP methods, score below GPT-4o on several benchmarks (e.g., 42.10% and 48.94% average), showing that fixed hand-designed models do not generalize.
- Gemini 2.0 Flash Thinking's 82.56% on MuMA-ToM is close to AutoToM's 81.44%, exceeding AutoToM on that single benchmark.
Limitations
- Requires a separate step to fuse multimodal inputs into text before inference.
- Model adjustment may fail to recognize relevant mental variables, leading to insufficient models.
- Higher inference cost from many LLM calls per question (e.g., ~27 API calls on average) and reliance on LLM-estimated conditionals.
Experimental setup
Datasets: ToMi BigToM MMToM-QA MuMA-ToM Hi-ToM FANToM OpenToM Online Watch-And-Help
Benchmarks: ToMi BigToM MMToM-QA MuMA-ToM Hi-ToM FANToM OpenToM Online Watch-And-Help
Metrics: accuracy Macro-F1 Pearson correlation speedup
Baselines: Llama 3.1 70B GPT-4o Gemini 2.0 Flash Gemini 2.0 Pro SymbolicToM SimToM DeepSeek-R1 Gemini 2.0 Flash Thinking o3-mini-high BIP-ALM LIMP
Builds on: Bayesian inverse planning BIP-ALM LIMP SymbolicToM SimToM I-POMDP
Closest papers by meaning
- Mental-Models for Multi-Agent Systems (NeurIPS 2026 · similarity 0.90)
- MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems (NeurIPS 2025 · similarity 0.90)
- Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action (NeurIPS 2026 · similarity 0.89)
- MuMA-ToM: Multi-modal Multi-Agent Theory of Mind (AAAI 2025 · similarity 0.89)
- Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner (ICML 2025 · similarity 0.89)
- Reflection-Bench: Evaluating Epistemic Agency in Large Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.