Research map / Vision-language and multimodal / Multimodal LLMs
Reality vs Counterfactual: Multi-World Contrastive Reinforcement Learning for Enhancing MLLM’s Theory of Mind in Egocentric Videos
Guiyang Hou, Yihui Fu, Chen Wu, Xiang Huang, Zhe Zheng, Wenqi Zhang, Yongliang Shen, Weiming Lu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a multi-world contrastive reinforcement learning approach that contrasts reality with counterfactual scenarios to enhance multimodal LLMs' theory of mind in egocentric videos.
Problem
Multimodal LLMs have weak theory of mind (inferring others' mental states) in egocentric video, which is needed for embodied agents to collaborate with humans.
Approach
Uses multi-world contrastive reinforcement learning, comparing reality against counterfactual worlds, to train MLLMs' ToM reasoning from egocentric video; details beyond the abstract are not available.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multi-world contrastive reward to GRPO | Qwen2.5-VL-7B GRPO · EgoToM in-domain | Average accuracy: 82.67% -> 84.67% | Improved |
| add multi-world contrastive reward to GRPO | Qwen2.5-VL-7B GRPO · EgoToM in-domain | Goal accuracy: 98.00% -> 94.00% | Hurt |
| add multi-world contrastive reward to GRPO | Qwen2.5-VL-7B GRPO · EgoTaskQA OOD | Average accuracy: 52.50% -> 57.50% | Improved |
| add multi-world contrastive reward to GRPO | Qwen2.5-VL-7B GRPO · EgoTaskQA OOD | Multi-hop accuracy: 53.33% -> 53.33% | No clear effect |
| RL with GRPO vs SFT | Qwen2.5-VL-7B · EgoToM in-domain | Average accuracy: 56.00% -> 82.67% | Improved |
| direct SFT on in-domain data | Qwen2.5-VL-7B · EgoToM in-domain | Average accuracy: 54.67% -> 56.00% | No clear effect |
| mixed multi-domain SFT | Qwen2.5-VL-7B · EgoToM in-domain | Average accuracy: 54.67% -> 56.67% | No clear effect |
| increase alpha from 0.3 to 0.6 (ablation) | MWCRL · EgoToM in-domain | Average accuracy: 84.67% -> 82.00% | Hurt |
| increase alpha from 0.3 to 0.6 (mu=1.0) (ablation) | MWCRL · EgoToM in-domain | Average accuracy: 84.00% -> 79.33% | Hurt |
| relax mu from 1.0 to 0.8 (ablation) | MWCRL (alpha=0.3) · EgoToM in-domain | Average accuracy: 84.00% -> 84.67% | Improved |
| relax mu from 1.0 to 0.6 (ablation) | MWCRL (alpha=0.3) · EgoToM in-domain | Average accuracy: 84.00% -> 85.33% | Improved |
Closest papers by meaning
- Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
- HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs (AAAI 2026 · similarity 0.87)
- SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions (NeurIPS 2025 · similarity 0.87)
- Mental-Models for Multi-Agent Systems (NeurIPS 2026 · similarity 0.86)
- Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models (ICLR 2025 · similarity 0.86)
- Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
