Research map / Vision-language and multimodal / Multimodal LLMs
Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEPM, a training-free inference-time method for MLLM emotion recognition combining confidence-guided coarse-to-fine classification with focus-on-emotion visual token dropping.
Problem
MLLMs struggle with emotion classification, confusing semantically similar emotion categories and being distracted by redundant visual information; existing fixes need fine-tuning or manual annotation.
Approach
Two-stage inference: a coarse positive/negative classification with logit-variance confidence guides a fine-grained emotion query, while visual tokens least attended under a "focus on emotion" prompt are dropped before the fine-grained stage.
Key results
- SEPM with LLaVA-7b (20% drop) improves average accuracy from 34.04 to 39.92 over zero-shot across five emotion datasets, including +17.19 on WebEmo7.
- With VILA-8b, SEPM (10% drop) raises average accuracy from 33.83 to 38.37.
- FoE-related token dropping outperforms random and query-related dropping on Emotion6 and EmoSet (e.g., 54.21 vs 53.20 random at β=0.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SEPM (β=0.2) instead of zero-shot inference | LLaVA-7b · Emotion6 | accuracy: 48.32 -> 54.04 | Helped |
| Apply SEPM (β=0.2) instead of zero-shot inference | LLaVA-7b · EmoSet | accuracy: 52.77 -> 56.24 | Helped |
| Apply SEPM (β=0.2) instead of zero-shot inference | LLaVA-7b · WebEmo7 | accuracy: 25.56 -> 42.75 | Helped |
| Apply SEPM (β=0.2) instead of zero-shot inference | VILA-8b · WebEmo7 | accuracy: 41.90 -> 39.10 | Hurt |
| Apply SEPM (β=0.2) instead of zero-shot inference | VILA-8b · Average | accuracy: 33.83 -> 38.01 | Helped |
| Use confidence-guided coarse-to-fine inference (ablation) | SEPM on LLaVA-7b · Emotion6 | accuracy: 48.32 -> 51.68 | Helped |
| Use confidence-guided coarse-to-fine inference (ablation) | SEPM on LLaVA-7b · EmoSet | accuracy: 52.77 -> 53.98 | Helped |
| Use Focus-on-Emotion prompts (ablation) | SEPM on LLaVA-7b · Emotion6 | accuracy: 48.32 -> 51.52 | Helped |
| Use Focus-on-Emotion prompts (ablation) | SEPM on LLaVA-7b · EmoSet | accuracy: 52.77 -> 54.28 | Helped |
| Add visual token augmentation to CCI and Focus-on-Emotion prompts (ablation) | SEPM on LLaVA-7b · Emotion6 | accuracy: 53.03 -> 54.04 | Helped |
| Add visual token augmentation to CCI and Focus-on-Emotion prompts (ablation) | SEPM on LLaVA-7b · EmoSet | accuracy: 56.10 -> 56.24 | Helped |
| Drop tokens using FoE relevance rather than random dropping (ablation) | LLaVA-7b · Emotion6 (β=0.1) | accuracy: 53.20 -> 54.21 | Helped |
Surprising findings
- SparseVLM, a token-pruning method, slightly improved emotion accuracy over zero-shot, unlike its typical drop on general tasks, suggesting emotion tasks contain more redundant visual tokens.
- Random token dropping reduces accuracy as drop rate increases, indicating visual tokens are not all redundant.
Limitations
- Gains on some datasets are small (e.g., WebEmo25 +2.55 for LLaVA, VILA on WebEmo7 drops at β=0.2).
- Only two MLLM backbones (LLaVA-7b and VILA-8b) are evaluated.
- Model preferences shift across emotion categories and the paper does not address per-category accuracy balance.
Experimental setup
Datasets: Emotion6 EmoSet WebEmo Abstract
Benchmarks: Emotion6 EmoSet WebEmo7 WebEmo25 Abstract
Metrics: accuracy
Baselines: Zero-shot Zero-shot-CoT SparseVLM Set-of-Vision (SoV)
Builds on: SparseVLM Zero-shot-CoT LLaVA VILA
Closest papers by meaning
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.92)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.91)
- Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier (AAAI 2026 · similarity 0.90)
- Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.90)
- MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.89)
- EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.