Research map / Video / Video-language understanding
VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VidEmo, a family of emotion-centric video foundation models trained with curriculum emotion learning and GRPO with an affective-tree reward, plus the 2.1M-sample Emo-CFG dataset, reporting gains over open and closed VideoLLMs on 15 face perception tasks.
Problem
Video LLMs struggle to understand complex, dynamic, context-dependent emotional states and to provide explainable rationales linking low-level facial attributes to high-level emotion.
Approach
Stage-wise curriculum emotion learning (attributes, then expressions, then emotions) followed by GRPO with a reward based on tree edit distance between hierarchical affective trees (attribute to expression to emotion) parsed from captions, plus search-based best-of-n reasoning at inference.
Key results
- VidEmo-T1 (7B) reaches 86.3% average on attribute perception, versus 80.6% for Qwen2.5-VL (7B), and 69.3% on emotion understanding, versus 63.6% for Qwen2.5-VL (7B).
- On DFEW, VidEmo reaches 64.92 UAR and 73.10 WAR, above EMO-LLaMA (60.23 / 65.89); on MAFW it reaches 44.02 UAR and 54.86 WAR.
- Ablation: the full model (CEL + ATR + ER) averages 67.0, versus 51.4 without components and 61.9 with CEL alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VidEmo-Base 3B instead of Qwen2.5-VL 3B | Emo-CFG overall evaluation · Emo-CFG, 1-3B overall average | accuracy: 46.1% -> 62.4%; +16.3% | Helped |
| use VidEmo-Base 7B instead of Qwen2.5-VL 7B | Emo-CFG overall evaluation · Emo-CFG, 7-8B overall average | accuracy: 51.7% -> 64.1%; +12.4% | Helped |
| use VidEmo-T1 7B instead of Qwen2.5-VL 7B | Emo-CFG attribute perception · Emo-CFG attribute perception | average score: +5.7% | Helped |
| use VidEmo-T1 7B instead of Qwen2.5-VL 7B | Emo-CFG expression analysis · Emo-CFG expression analysis | average score: +6.8% | Helped |
| use VidEmo-T1 7B instead of Gemini 2.0 | Emo-CFG emotion understanding · Emo-CFG emotion understanding | average score: 63.5% -> 69.3% | Helped |
| fine-tune VidEmo for the downstream expression task instead of using EMO-LLaMA | facial expression recognition · DFEW and MAFW | UAR and WAR: 7.8% in UAR and 10.9% in WAR on DFEW; 5.9% in UAR and 12.8% in WAR on MAFW | Helped |
| add curriculum emotion learning (CEL) (ablation) | VidEmo · Emo-CFG ablation benchmark | average: 51.4 -> 61.9 | Helped |
| add affective-tree reward (ATR) to CEL (ablation) | VidEmo · Emo-CFG ablation benchmark | average: 61.9 -> 63.6 | Helped |
| add emotion reasoning (ER) to CEL + ATR (ablation) | VidEmo · Emo-CFG ablation benchmark | average: 63.6 -> 67.0 | Helped |
| use multi-stage attribute and expression training rather than attribute-only single-stage training (ablation) | VidEmo curriculum emotion learning · Emo-CFG curriculum learning | average: 40.7 -> 52.6 | Helped |
| add the emotion-data stage to multi-stage attribute + expression training (ablation) | VidEmo curriculum emotion learning · Emo-CFG curriculum learning | average: 52.6 -> 61.9 | Helped |
| add Tree Reward to GRPO (ablation) | VidEmo post-training · Emo-CFG affective-tree reward | average: 51.2 -> 61.4 | Helped |
Surprising findings
- Closed-source Gemini 2.0 scores only 26.3% on fine-grained sentiment analysis according to the introduction, while the open 7B VidEmo model exceeds closed APIs on all three task categories.
- Attribute perception average is higher than expression and emotion averages, and perplexity increases from attribute to expression to emotion.
Limitations
- Susceptible to counterfactual or emotionally inconsistent generations, as with most VideoLLMs.
- Does not incorporate audio or other contextual modalities.
Experimental setup
Datasets: Emo-CFG CelebV-HQ CelebV-Text MEAD MOSEI MOSI CH-SIMS MAFW DFEW MER2023 RAVDESS CASME II
Benchmarks: Emo-CFG benchmark DFEW MAFW
Metrics: accuracy F1 GPT score UAR WAR
Baselines: Gemini 2.0 Claude 3.5 Sonnet Qwen-VL-MAX GPT-4o GPT-4o mini LLaVA-OV InternVL2.5 VideoLLaMA3 mPLUG-Owl3 Qwen2.5-VL EMO-LLaMA DFER-CLIP
Builds on: GRPO Qwen2.5-VL LLaVA-CoT
Closest papers by meaning
- EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning (CVPR 2024 · similarity 0.89)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.88)
- EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
- Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis (AAAI 2026 · similarity 0.87)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.87)
- EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessment (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.