AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

Reality vs Counterfactual: Multi-World Contrastive Reinforcement Learning for Enhancing MLLM’s Theory of Mind in Egocentric Videos

Guiyang Hou, Yihui Fu, Chen Wu, Xiang Huang, Zhe Zheng, Wenqi Zhang, Yongliang Shen, Weiming Lu

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a multi-world contrastive reinforcement learning approach that contrasts reality with counterfactual scenarios to enhance multimodal LLMs' theory of mind in egocentric videos.

Problem

Multimodal LLMs have weak theory of mind (inferring others' mental states) in egocentric video, which is needed for embodied agents to collaborate with humans.

Approach

Uses multi-world contrastive reinforcement learning, comparing reality against counterfactual worlds, to train MLLMs' ToM reasoning from egocentric video; details beyond the abstract are not available.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add multi-world contrastive reward to GRPOQwen2.5-VL-7B GRPO · EgoToM in-domainAverage accuracy: 82.67% -> 84.67%Improved
add multi-world contrastive reward to GRPOQwen2.5-VL-7B GRPO · EgoToM in-domainGoal accuracy: 98.00% -> 94.00%Hurt
add multi-world contrastive reward to GRPOQwen2.5-VL-7B GRPO · EgoTaskQA OODAverage accuracy: 52.50% -> 57.50%Improved
add multi-world contrastive reward to GRPOQwen2.5-VL-7B GRPO · EgoTaskQA OODMulti-hop accuracy: 53.33% -> 53.33%No clear effect
RL with GRPO vs SFTQwen2.5-VL-7B · EgoToM in-domainAverage accuracy: 56.00% -> 82.67%Improved
direct SFT on in-domain dataQwen2.5-VL-7B · EgoToM in-domainAverage accuracy: 54.67% -> 56.00%No clear effect
mixed multi-domain SFTQwen2.5-VL-7B · EgoToM in-domainAverage accuracy: 54.67% -> 56.67%No clear effect
increase alpha from 0.3 to 0.6 (ablation)MWCRL · EgoToM in-domainAverage accuracy: 84.67% -> 82.00%Hurt
increase alpha from 0.3 to 0.6 (mu=1.0) (ablation)MWCRL · EgoToM in-domainAverage accuracy: 84.00% -> 79.33%Hurt
relax mu from 1.0 to 0.8 (ablation)MWCRL (alpha=0.3) · EgoToM in-domainAverage accuracy: 84.00% -> 84.67%Improved
relax mu from 1.0 to 0.6 (ablation)MWCRL (alpha=0.3) · EgoToM in-domainAverage accuracy: 84.00% -> 85.33%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy