Research map / Vision-language and multimodal / Multimodal LLMs

EmoThinker: Advancing Visual-Acoustic Emotion Analysis via Structural Token Selection and Chain-of-Thought Reasoning

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EmoThinker, a Qwen2.5-Omni-based multimodal emotion analysis model with structural visual token selection (face focus plus conditioned background), a text-conditioned audio evidence extractor, and CoET chain-of-thought training on a newly built CoET dataset.

Problem

Multimodal emotion analysis struggles with sparse, localized emotional cues across video and audio and with temporal asynchrony between modalities, which implicit fusion entangles and dilutes.

Approach

Decompose video frames into face (focus) and background regions, condition background tokens on the text query with cross-attention and randomly drop them, compress audio via text-guided cross-attention, and train the LLM with explicit step-by-step chain-of-emotion-thought reasoning traces.

Key results

• On DFEW fine-tuned setting, EmoThinker reaches 65.23 UAR and 78.13 WAR, versus 64.21 UAR and 77.06 WAR for Emotion-LLaMA.
• On DFEW zero-shot, EmoThinker achieves 51.08 UAR and 65.63 WAR, about 10.5 WAR points above Emotion-LLaMA (59.37).
• On EMER reasoning, EmoThinker scores 8.67 clue overlap and 7.53 label overlap, above Emotion-LLaMA (8.22, 6.25).

Surprising findings

• Random dropping of background tokens outperforms attention-based token selection for background pruning.
• Increasing the background token dropping ratio degrades performance, so the authors keep a low drop ratio despite computational savings.

Limitations

• Performance on minority classes such as Sad and Fear is unstable due to class imbalance.
• Dropping more background tokens degrades accuracy, so efficiency gains trade off against performance.
• Reasoning evaluation relies on ChatGPT-based scoring rather than human judgment.

Experimental setup

Datasets: CoET DFEW MUStARD UR-FUNNY IEMOCAP MELD EMER

Benchmarks: DFEW IEMOCAP MELD MUStARD UR-FUNNY EMER

Metrics: UAR WAR weighted-F1 accuracy clue overlap label overlap

Baselines: Emotion-LLaMA Qwen2.5-Omni Qwen2.5-VL MiniGPT-v2 Video-LLaVA MultiEMO MAE-DFER VideoMAE GPT-4V

Builds on: Qwen2.5-Omni Emotion-LLaMA LoRA

Closest papers by meaning

• Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.88)
• AVERE: Improving Audiovisual Emotion Reasoning with Preference Optimization (ICLR 2026 · similarity 0.88)
• Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models (ICML 2025 · similarity 0.88)
• EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessment (ICML 2026 · similarity 0.88)
• Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier (AAAI 2026 · similarity 0.88)
• Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.87)