Research map / Large language models / Agents, code and math
Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a scalable Bayesian inverse-planning ToM planner in which small post-trained Llama LMs steer larger (70B/405B) frozen LMs at inference time via a weak-to-strong logit-ratio redirection, evaluated on MMToM-QA.
Problem
Multimodal theory-of-mind reasoning over multi-step agent plans degrades as planning steps grow, and small LMs lack the world knowledge needed for ToM likelihood estimation, so existing ToM workflows and fine-tuning do not scale.
Approach
Decompose ToM inference into stepwise Bayesian inverse-planning updates whose action likelihoods come from a large LM, and transfer ToM-specific behaviour from a small post-trained LM to the large LM at inference via a logit-ratio redirection (weak-to-strong control), avoiding post-training of the large model.
Key results
- The 405B weak-to-strong method (8B to 405B) reaches 81.3% accuracy on multimodal MMToM-QA, reported as a 4.6% gain over the best prior baseline.
- Zero-shot Llama3.1-405B reaches 69.43% on the Llama3.1 scaling table, above 8B (65.19%) and 70B (66.62%) zero-shot.
- Weak-to-strong control with a 4B-width controller on 70B reaches 78.52% overall, close to the 8B-controlled variant (79.38%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply the scaled Bayesian planner with weak-to-strong control and multimodal inputs | Llama3.1-405B · MMToM-QA multimodal | accuracy (all): 76.7 -> 81.3 | Helped |
| replace weak-to-strong control with naive logit addition (ablation) | Llama3.1-70B with an 8B controller · MMToM-QA | accuracy (all): 79.38 -> 74.45 | Hurt |
| replace weak-to-strong control with naive logit addition (ablation) | Llama3.1-405B with an 8B controller · MMToM-QA | accuracy (all): 81.29 -> 74.67 | Hurt |
| increase the controlled model from Llama3.1-70B to Llama3.1-405B (ablation) | weak-to-strong Bayesian planner · MMToM-QA | accuracy (all): 79.38 -> 81.29 | Helped |
| post-train Llama3.1-8B instead of using it zero-shot (ablation) | Llama3.1-8B · MMToM-QA | accuracy (all): 65.19 -> 75.71 | Helped |
| use a post-trained 8B controller to guide 70B rather than use the post-trained 8B alone (ablation) | Llama3.1 weak-to-strong planner · MMToM-QA | accuracy (all): 75.71 -> 79.38 | Helped |
| reduce controller size from 8B to 4B-Width while guiding 70B (ablation) | Llama3.1 weak-to-strong planner · MMToM-QA | accuracy (all): 79.38 -> 78.52 | Hurt |
| use a 4B-Width controller instead of a 4B-Depth controller to guide 70B (ablation) | Llama3.1 weak-to-strong planner · MMToM-QA | accuracy (all): 78.38 -> 78.52 | Helped |
| use full fine-tuning instead of LoRA (ablation) | GPT2-large · MMToM-QA, 20,000 training points | accuracy: 62.4 -> 63.4 | Helped |
| use full fine-tuning instead of LoRA (ablation) | Gemma-2B · MMToM-QA, 20,000 training points | accuracy: 68.5 -> 68.8 | Helped |
| use full fine-tuning instead of LoRA for the 4B-Width controller (ablation) | 4B-Width controller guiding Llama3.1-70B · MMToM-QA | accuracy: 78.52 -> 78.67 | Helped |
Surprising findings
- Naively adding small-LM logits to large-LM logits performs worse than the structured weak-to-strong control.
- Reducing the weak controller from 8B to 4B kept performance nearly unchanged, and the width-reduced 4B controller transferred better than the depth-reduced one.
Limitations
- Gains are shown mostly on MMToM-QA and synthetic VirtualHome-style scenarios, so real-world generalization is untested.
- The theoretical KL bound relies on a small-step gradient approximation and a loose Delta-s assumption, which the paper itself acknowledges is approximate.
- Several reported comparisons vary in configuration and are sensitive to hyperparameters and topic phrasing (variance ranges of about 1.6 to 3.9 points).
Experimental setup
Datasets: MMToM-QA MMToM (VirtualHome procedural data)
Benchmarks: MMToM-QA unseen themed scenarios (Andersen tales, ancient Egyptian, outer space, wild west, medieval castle)
Metrics: accuracy
Baselines: GPT-4 GPT-4V SimToM SymbolicToM BIP-ALM DeepSeek-R1-671B OpenAI-o3-mini Video-Llama2-13B LLaVA-7B
Builds on: Bayesian inverse planning DPO BIP-ALM proxy tuning MMToM-QA
Closest papers by meaning
- Unveiling the Cognitive Compass: Theory-of-Mind–Guided Multimodal Emotion Reasoning (ICLR 2026 · similarity 0.89)
- Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
- From Black Boxes to Transparent Minds: Evaluating and Enhancing the Theory of Mind in Multimodal Large Language Models (ICML 2025 · similarity 0.89)
- AutoToM: Scaling Model-based Mental Inference via Automated Agent Modeling (NeurIPS 2025 · similarity 0.89)
- Evolving Minds: Logic-Informed Inference from Temporal Action Patterns (ICML 2025 · similarity 0.88)
- Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.