Research map / Vision-language and multimodal / Multimodal LLMs

Unveiling the Cognitive Compass: Theory-of-Mind–Guided Multimodal Emotion Reasoning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HitEmotion, a ToM-grounded three-level multimodal emotion benchmark built from 24 re-purposed datasets, and TMPO, an SFT-then-GRPO method with structure, content, process, and consistency rewards on Qwen2.5-Omni-7B.

Problem

Current MLLMs have limited deep emotional understanding, and existing emotion benchmarks lack a cognitive (Theory-of-Mind) hierarchy that pinpoints where reasoning breaks down.

Approach

Organize emotion tasks into three ToM-aligned cognitive levels and train the model to emit structured ToM reasoning chains via SFT followed by GRPO with a multi-dimensional reward that includes ToM-keyword and consistency terms.

Key results

• TMPO (Qwen2.5-Omni-7B + GRPO) tops 16 of 24 HitEmotion tasks, per the paper, and reaches 77.12 on FESD vs. 78.39 for Gemini-2.5-Pro on that task.
• Adding a ToM prompt improves GPT-4.1 on Emotion Interpretation from 57.67 to 69.00 and Gemini-2.5-Pro on SFA from 41.22 to 52.78.
• On the ECR level, TMPO+GRPO reaches 73.13 on EER and 72.27 on EI, and no evaluated model exceeds 60 average on the ECR level except as noted by the paper's own claim.

Surprising findings

• ToM prompting degrades weaker models; VideoLLaMA3-7B drops from 61.78 to 54.18 on FESD.
• Removing the structure reward causes format collapse, so the output XML schema is not followed.

Limitations

• Level 1 perception performance is bounded by the base model's sensory encoders.
• Restricted to 7B omni-modal backbone due to lack of larger open audio-visual models.
• TMPO targets social reasoning where objective verification is hard, so reward design relies on LLM-based consistency judgments.

Experimental setup

Datasets: CH-SIMS CH-SIMSv2 EmoSet Memotion MER2023 CMU-MOSI CMU-MOSEI RAVDESS FMSA-SC SemEval-2021 Task 6 MC-EIU UR-FUNNY

Benchmarks: HitEmotion

Metrics: ACC micro F1 weighted F1 exact match F1 LLM-judged score

Baselines: GPT-4o GPT-4.1 Gemini-2.5-Flash Gemini-2.5-Pro Qwen2.5-VL-7B Qwen2.5-VL-32B InternVL3-8B InternVL3-38B Emotion-LLaMA-7B AffectGPT-7B HumanOmni-7B R1-Omni-0.5B

Builds on: GRPO Chain-of-Thought Qwen2.5-Omni

Closest papers by meaning

• EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
• MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.89)
• Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner (ICML 2025 · similarity 0.89)
• Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier (AAAI 2026 · similarity 0.89)
• From Stimuli to Minds: Enhancing Psychological Reasoning in LLMs via Bilateral Reinforcement Learning (AAAI 2026 · similarity 0.89)
• From Black Boxes to Transparent Minds: Evaluating and Enhancing the Theory of Mind in Multimodal Large Language Models (ICML 2025 · similarity 0.89)