Research map / Vision-language and multimodal / Multimodal LLMs
EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EmoVIT, an emotion-specific visual instruction tuning model built on InstructBLIP, trained on GPT-4-generated emotion instruction data (categorical, conversation, reasoning) over EmoSet images.
Problem
Visual instruction-tuned large multimodal models perform poorly on visual emotion recognition, and there is no emotion-specific instruction data for training them.
Approach
A GPT-4-assisted pipeline that generates instance-wise emotion instructions from captions and low/mid/high-level emotion attributes (brightness, colorfulness, scene, objects, facial expression, action), used to fine-tune the InstructBLIP Q-Former while keeping the image encoder and LLM frozen.
Key results
- On EmoSet held-in test, EmoVIT reaches 83.36% accuracy vs 42.20% for InstructBLIP baseline.
- Held-out, EmoVIT achieves 68.09% on FI and 57.81% on Emotion6, outperforming InstructBLIP (37.97%, 46.11%).
- On UnbiasedEmo, EmoVIT reaches 74.72% vs 74.27% for curriculum learning, the best prior method in the table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune InstructBLIP on emotion visual instruction data | InstructBLIP · EmoSet test | accuracy (%): 46.79 -> 83.36 | Helped |
| fine-tune InstructBLIP on emotion visual instruction data | InstructBLIP · FI test | accuracy (%): 57.72 -> 68.09 | Helped |
| fine-tune InstructBLIP on emotion visual instruction data | InstructBLIP · Emotion6 test | accuracy (%): 50.00 -> 57.81 | Helped |
| fine-tune InstructBLIP on emotion visual instruction data | InstructBLIP · UnbiasedEmo test | accuracy (%): 74.27 -> 74.72 | Helped |
| add Categorical instruction data (ablation) | InstructBLIP · EmoSet test | accuracy (%): 42.20 -> 80.90 | Helped |
| add Conversation instruction data to Categorical data (ablation) | InstructBLIP · EmoSet test | accuracy (%): 80.90 -> 81.95 | Helped |
| add Reasoning instruction data to Categorical and Conversation data (ablation) | InstructBLIP · EmoSet test | accuracy (%): 81.95 -> 83.36 | Helped |
| vary the pretraining data portion from 5% to 50% (ablation) | EmoVIT · EmoSet test | accuracy (%): 79.00; 81.00; 79.34; 83.36 | Mixed |
| replace Vicuna-7B with Vicuna-13B (ablation) | EmoVIT · EmoSet test | accuracy (%): 83.36 -> 82.21 | Hurt |
| replace Vicuna-7B with FlanT5XL (ablation) | EmoVIT · EmoSet test | accuracy (%): 83.36 -> 80.98 | Hurt |
| use GPT-4 Turbo instead of GPT-4 to generate conversational data (ablation) | EmoVIT · EmoSet test | accuracy (%): 83.36% vs 82.96% | No clear effect |
| add three in-context caption/emotion examples per category during held-out evaluation (ablation) | EmoVIT · held-out evaluation | performance | No clear effect |
Surprising findings
- A smaller Vicuna-7B LLM outperformed Vicuna-13B and FlanT5-XL in the authors' ablation, which they attribute to limited training data.
- Generated humor captions were preferred by 30 participants 60% of the time versus human-written captions.
Limitations
- Held-in pretraining used less than 50% of EmoSet due to GPT-API cost.
- Held-out evaluation is not strict zero-shot since the dataset is emotion-only.
- Larger LLMs did not improve results, possibly due to limited training data.
Experimental setup
Datasets: EmoSet WEBEmo Emotion6 FI ArtPhoto IAPS Abstract EmotionROI UnbiasedEmo OxfordTVG-HIC
Benchmarks: EmoSet WebEmo FI Emotion6 Abstract ArtPhoto IAPSa EmotionROI UnbiasedEmo OxfordTVG-HIC
Metrics: accuracy sensitivity user preference vote
Baselines: Flamingo LLaVA BLIP-2 InstructBLIP Direct Learning Self-Directed Learning Joint Learning Curriculum Learning
Builds on: InstructBLIP LLaVA BLIP-2 LAVIS
Closest papers by meaning
- VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models (NeurIPS 2025 · similarity 0.89)
- EmoEdit: Evoking Emotions through Image Manipulation (CVPR 2025 · similarity 0.89)
- EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.87)
- EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessment (ICML 2026 · similarity 0.87)
- Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.