Research map / Vision-language and multimodal / Multimodal LLMs

Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Emotion Statement Judgment (ESJ) and the INSETS pipeline to automatically build emotion-centric true/false statements, releasing INSETS-462k and the human-refined MVEI benchmark for evaluating MLLMs' visual emotion perception.

Problem

Existing visual emotion evaluations (classification and interpretation with fixed ground-truth answers, limited taxonomies, no context, costly crowdsourced labels) are ill-suited to MLLMs, producing inconsistent findings on their emotion perception ability.

Approach

Reformulate visual emotion evaluation as judging the correctness of emotion-centric statements over four dimensions (sentiment polarity, emotion interpretation, scene context, perception subjectivity), with open-vocabulary labels assigned by majority vote across multiple MLLMs and statements auto-constructed with minimal human refinement.

Key results

• GPT4o achieves the highest overall MVEI accuracy among evaluated MLLMs at 78.3%, while 25 human participants average 91.6% on a 300-sample subset.
• Qwen2.5-VL fine-tuned fully (freeze vision) on 10k INSETS samples improves total accuracy from 75.9% to 81.9%; scaling to 459k reaches 83.4%.
• Perception subjectivity remains the weakest dimension for most models, with only modest gains from adaptation (66.3% to 73.3% for Qwen2.5-VL).

Surprising findings

• Perception subjectivity accuracy improves only modestly under in-context learning, LoRA, full fine-tuning, and GRPO, suggesting it depends on inherent model properties rather than task adaptation.
• Several older MLLMs (InstructBLIP, Otter) perform near or below chance with strong response biases, and the paper finds current MLLMs not yet reliable as LLM-as-a-judge for affective perception.

Limitations

• Evaluation limited to MLLMs under 10B parameters due to hardware constraints.
• Monolingual (English) evaluation only.
• Ethics statement notes positive-polarity skew (65.2%) and possible demographic and cultural biases in generated characters and labels.

Experimental setup

Datasets: EmoSet INSETS-462k

Benchmarks: MVEI

Metrics: accuracy positive ratio give-up ratio

Baselines: BLIP2 InstructBLIP Otter DeepSeek-VL Paligemma MiniCPM GPT4o-mini GPT4o

Builds on: Parrott's hierarchical emotion model EmoSet GRPO LoRA

Closest papers by meaning

• Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models (ICML 2025 · similarity 0.92)
• MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.92)
• AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.92)
• EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessment (ICML 2026 · similarity 0.89)
• Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.89)
• II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.89)