Research map / Vision-language and multimodal / Multimodal LLMs

Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an Emotional Rationale Verifier (a RoBERTa classifier distilled from GPT-4.1 pseudo-labels) whose sentence-level, neutral-filtered reward is added to GRPO training of HumanOmni to make multimodal emotion explanations consistent with predicted and target emotions.

Problem

MLLM-based multimodal emotion recognition produces explanations that diverge from target labels or contradict the predicted emotion, and existing RL rewards only check the final label, not rationale coherence.

Approach

A lightweight verifier (ERV) fine-tuned from GoEmotions RoBERTa on GPT-4.1 pseudo-labeled explanation sentences scores each generated explanation sentence by emotion; the explanation reward is the proportion of non-neutral sentences matching the ground-truth emotion, added to format and answer rewards in GRPO.

Key results

• On MAFW, the 0.5B model improves EEA from 32.41% (baseline) to 48.93% and FCR from 27.27% to 43.57%; the 7B variant reaches 54.70% EEA and 50.98% FCR.
• On DFEW, the 0.5B model reaches 59.45% EEA and 53.75% FCR versus 41.66% and 36.39% for the baseline.
• In a human study with 20 raters on 28 samples, the method won 53.6% of comparisons against R1-Omni with a higher mean score (3.52 vs 3.42).

Surprising findings

• GRPO with only answer and format rewards lowered explanation-prediction consistency (EPC) relative to SFT (e.g., 40.98% baseline vs 64.30% EMER-SFT on DFEW-level comparisons), indicating RL focused on the answer degrades rationale consistency.
• R1-Omni reached higher WAR/UAR than the proposed 0.5B model on MAFW, while the proposed method traded off some accuracy for consistency.

Limitations

• Evaluation of explanation emotion relies on a closed-source LLM judge (GPT-4.1-mini), with only a single additional judge checked.
• Training data for the verifier is pseudo-labeled by GPT-4.1 from R1-Omni outputs, so label noise and the teacher's biases propagate.
• Classification accuracy (WAR/UAR) is not uniformly improved; R1-Omni is better on UAR for MAFW and on several accuracy measures.

Experimental setup

Datasets: EMER MERR-Fine DFEW MAFW GoEmotions

Benchmarks: MAFW DFEW

Metrics: EEA FCR EPC WAR UAR

Baselines: EMER-SFT R1-Omni Emotion-LLaMA Baseline (SFT+GRPO with answer and format rewards)

Builds on: GRPO HumanOmni EMER R1-Omni GoEmotions

Closest papers by meaning

• MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.91)
• Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models (ICML 2025 · similarity 0.90)
• Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.90)
• AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.89)
• Unveiling the Cognitive Compass: Theory-of-Mind–Guided Multimodal Emotion Reasoning (ICLR 2026 · similarity 0.89)
• EmotionThinker: Prosody-Aware Reinforcement Learning for Explainable Speech Emotion Reasoning (ICLR 2026 · similarity 0.89)