Research map / Vision-language and multimodal / Multimodal LLMs

Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VisionToM, a backbone-frozen intervention method that steers attention heads of video MLLMs using PGD-adversarial visual negatives and clustered ToM-reasoning encoders, improving video-only Theory of Mind QA on EgoToM.

Problem

Multimodal LLMs perform poorly at Theory of Mind inference from video-only egocentric input and are prone to hallucination driven by linguistic priors; existing ToM interpretability interventions are text-only.

Approach

Compute intervention vectors from attention-head activations: a visual-attention offset from PGD-adversarial vs. clean frames, plus per-cluster learned encoders that map negative ToM-reasoning representations toward positives, then add the combined direction to top-K heads at inference with the MLLM frozen.

Key results

• On EgoToM, VisionToM (+alpha Delta) raises LLaVA-Next-Video-7B from 61.5% to 74.5% on Goal, 38.9% to 45.3% on Belief, and 24.0% to 29.7% on Actions.
• Qwen2.5-VL-7B with VisionToM reaches 88.9% Goal (matching the human 88-89% range), 42.0% Belief, and 37.6% Actions versus baseline 86.9/35.6/31.1.
• On Qwen2.5-VL-72B, VisionToM improves Belief from 43.6% to 59.8% and Actions from 37.1% to 57.3%.

Surprising findings

• Random Gaussian noise negatives provide weak direction, while PGD adversarial negatives yield substantially larger gains.
• Intervention strength beyond a range (e.g., alpha=4 for LLaVA) causes invalid outputs such as garbled or repeating text.

Limitations

• Gains are reported on a small benchmark (EgoToM, about 350 samples per task) with a single calibration split.
• Intervention strength must be kept within a narrow range or outputs become invalid.
• Open-ended evaluation relies on DeepSeek-R1 judges, with human agreement checked only on a manual subset.

Experimental setup

Datasets: EgoToM Ego4D MMToM-QA

Benchmarks: EgoToM MMToM-QA

Metrics: Top-1 accuracy True % Info % True and Info %

Baselines: GPT-4-Turbo GPT-4o Gemini-2.5-Flash VideoLLaMA2-72B CogVLM2 GLM-4.1V-9B-Thinking Random noise delta (Rnd-Delta) GridToM

Builds on: GridToM TruthfulQA EgoToM

Closest papers by meaning

• From Black Boxes to Transparent Minds: Evaluating and Enhancing the Theory of Mind in Multimodal Large Language Models (ICML 2025 · similarity 0.93)
• Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens (CVPR 2026 · similarity 0.90)
• EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models (CVPR 2024 · similarity 0.90)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)
• BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of Mind (AAAI 2024 · similarity 0.89)
• HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs (AAAI 2026 · similarity 0.89)