Research map / Vision-language and multimodal / Multimodal LLMs

Reinforce Trustworthiness in Multimodal Emotional Support System

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MultiMood, a multimodal (video, audio, text) emotional-support LLM trained with SFT on MESC and then RL (GRPO/PPO) with trustworthiness and similarity rewards, evaluated on MESC and DFEW.

Problem

Existing emotional support systems are mostly text-only or only do emotion recognition, and response generation often lacks trustworthiness and accurate identification of emotional support elements.

Approach

Combine modality-specific encoders (SigLIP video, BEATs audio) and a Mamba-based conversation compressor with an LLM, then align responses to seven psychology-derived trustworthiness dimensions via RL where the reward averages a GPT-4o trust score and a BGE-M3 ColBERT similarity to reference responses.

Key results

• On DFEW, MultiMood reports UAR 85.94% and WAR 89.89%, above the prior best EmotionLLaMA (64.21 UAR, 77.06 WAR).
• On MESC response generation, MultiMood SFT+GRPO reaches BLEU-2 6.18, ROUGE-L 17.86, BERTScore 86.80, the best among compared models.
• Human evaluation favors MultiMood SFT+RL over Qwen2-7B baseline (overall 58% vs 21%), and GPT-4o judge average rises from 5.4 (SFT) to 6.8 (SFT+RL).

Surprising findings

• Strategy prediction accuracy is lower than BlenderBot SFT (48%) and SMES (49%), with MultiMood at 42.81%.
• System emotion prediction accuracy exceeds 90% for most models, attributed to 90% Neutral labels in the data.

Limitations

• Underperforms BlenderBot and SMES on strategy prediction due to class imbalance and lack of retrieval.
• Uses GPT-4o as the trust reward, which may introduce bias into RL training.
• Datasets come from movies and TV drama rather than real therapy sessions.

Experimental setup

Datasets: MESC DFEW

Benchmarks: MESC DFEW

Metrics: Accuracy Weighted-F1 BLEU-2 ROUGE-L BERTScore UAR WAR human fluency/comfort/overall win rate LLM judge trust scores

Baselines: SMES EmotionLLaMA VideoLLaVA InternVideo2.5-8B VideoLLaMA3-7B GPT-4o Claude-3.7-Sonnet Deepseek-R1 Qwen2.5-7B BlenderBot SFT

Builds on: VideoLLaMA2 GRPO PPO TrustLLM ECoT

Closest papers by meaning

• Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier (AAAI 2026 · similarity 0.88)
• MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.88)
• EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models (CVPR 2026 · similarity 0.87)
• Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy (CVPR 2026 · similarity 0.86)
• Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.86)
• Multi-to-Single: Reducing Multimodal Dependency in Emotion Recognition Through Contrastive Learning (AAAI 2025 · similarity 0.86)