Research map / Vision-language and multimodal / Multimodal LLMs

MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MME-Emotion, a video benchmark of ~6,500 QA pairs across 8 emotion/sentiment/intent tasks, with an automated multi-agent MLLM-as-judge suite scoring recognition, reasoning, and CoT, evaluated on 20 MLLMs.

Problem

Existing emotional intelligence benchmarks for MLLMs cover limited scenarios, use inconsistent protocols, and only measure emotion recognition rather than the reasoning behind emotional states.

Approach

A unified closed-set QA benchmark over diverse public emotion datasets, with an annotation-free multi-agent judge (GPT-4.1 step extraction, Qwen2-Audio audio clues, GPT-4o judge) that scores each reasoning step, yielding Rec-S, Rea-S and CoT-S metrics validated against five human experts.

Key results

• Gemini-2.5-Pro achieves the top CoT-S of 56.0% and Rec-S of 39.3%; average across models is 29.4% Rec-S and 39.5% CoT-S.
• Audio-Reasoner (audio+text only, 7B) reaches 54.8% CoT-S, close to GPT-4o's 53.8%, while omnimodal models often underperform.
• GPT-judge vs. five human experts: Spearman 0.9530, Cohen's kappa 0.8626, ICC 0.9704 on 373 reasoning steps.

Surprising findings

• Audio-only Audio-Reasoner rivals top vision-language models, and omnimodal models often show noticeable performance drops, suggesting poor multimodal fusion.
• Emotion-specialist models such as Emotion-LLaMA and HumanOmni score near zero on reasoning score because they answer without explicit reasoning.

Limitations

• No task difficulty levels are assigned to samples.
• Multilingual differences are not analyzed.
• Judge relies on GPT-4o and an audio agent, so evaluation depends on proprietary and intermediate model quality.

Experimental setup

Datasets: multiple public emotion video datasets (test sets)

Benchmarks: MME-Emotion

Metrics: recognition score (Rec-S) reasoning score (Rea-S) chain-of-thought score (CoT-S) average step count average token count

Baselines: Qwen2-Audio Audio-Reasoner Qwen2-VL-72B Qwen2.5-VL-72B QVQ Video-LLaMA2 Qwen2.5-Omni Emotion-LLaMA HumanOmni R1-Omni AffectGPT GPT-4o

Builds on: Audio-Reasoner R1-Omni AffectGPT Emotion-LLaMA HumanOmni

Closest papers by meaning

• Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.92)
• AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.91)
• Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier (AAAI 2026 · similarity 0.91)
• Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy (CVPR 2026 · similarity 0.90)
• Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.90)
• Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models (ICML 2025 · similarity 0.89)