Research map / Video / Video-language understanding
$E^3$: Exploring Embodied Emotion Through A Large-Scale Egocentric Video Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces E3, a ~50-hour egocentric (first-person) video dataset with emotion annotations and four benchmark tasks, plus Emotion-LlaMa, a MiniGPT4-Video-based MLLM extended with an AudioMAE audio branch.
Problem
Emotion analysis has mostly used third-person videos, leaving embodied agents without first-person emotion data that captures the camera wearer's emotional context.
Approach
Collect a large first-person vlog dataset with multi-stage manual annotations (emotion intensity, per-person emotion categories including hidden emotions, start/end localization, and reasoning answers), and extend a video MLLM (MiniGPT4-Video) with an audio branch via AudioMAE to fuse acoustic cues with visual and subtitle tokens.
Key results
- Emotion-LlaMa achieves Acc 85.93 and F1 93.70 on embodied emotion recognition, above the fine-tuned MINI (83.83/92.64).
- On emotion classification Emotion-LlaMa reaches avg F1 45.62, versus 44.95 for fine-tuned MINI and 43.55 for fine-tuned VAST.
- On emotion reasoning Emotion-LlaMa scores CIDEr 84.06 and BLEU-4 15.99, above fine-tuned MINI (81.22 CIDEr) and VAST (75.51 CIDEr).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add an audio feature branch to MiniGPT4-Video | Emotion-LlaMa · Embodied emotion recognition test | Acc: 84.92 -> 85.93 | Helped |
| Add an audio feature branch to MiniGPT4-Video | Emotion-LlaMa · Embodied emotion reasoning test | IC: 2.22 -> 2.39 | Helped |
| Add subtitle text (ablation) | Emotion-LlaMa · Embodied emotion reasoning test | IC: 2.25 -> 2.39 | Helped |
| Add audio (ablation) | Emotion-LlaMa · Embodied emotion reasoning test | IC: 2.22 -> 2.39 | Helped |
| Add subtitle text and audio (ablation) | Emotion-LlaMa · Embodied emotion reasoning test | IC: 2.03 -> 2.39 | Helped |
| Increase the fine-tuning data volume (ablation) | MLLMs on E3 · Embodied emotion analysis | performance | Helped |
Surprising findings
- Fine-tuning MALMM decreased its Micro-F1 on emotion classification, attributed to its memory bank storing reference histories that contradict target emotions.
- Task-specific models trained on E3 underperform zero-shot MLLMs on emotion recognition, which the authors attribute to limited parameter count.
Limitations
- Videos are drawn from vlogs on YouTube, BiliBili and Douyin, so the distribution may be skewed toward social-media content.
- Emotion localization and reasoning results for LLM-based models lack multiple candidates and show temporal comprehension and hallucination issues.
Experimental setup
Datasets: E3
Benchmarks: E3
Metrics: Accuracy F1 Micro-F1 mIoU R1@IoU R5@IoU BLEU-3 BLEU-4 ROUGE-L CIDEr
Baselines: MMIM CENET ALMT MALMM VAST MINI DFAN VAANet CTEN MSAT r2-tuning VTG-LLM
Builds on: MiniGPT4-Video AudioMAE EVA-CLIP LLaMA
Closest papers by meaning
- EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition (NeurIPS 2025 · similarity 0.88)
- Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.87)
- EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessment (ICML 2026 · similarity 0.87)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.87)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.87)
- Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.