Research map / Image generation / Autoregressive and latent generation
FindingEmo: An Image Dataset for Emotion Recognition in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FindingEmo, a 25k-image dataset of multi-person social scenes annotated for whole-scene valence, arousal, and Plutchik emotion, and benchmarks CNN and ViT baselines on it.
Problem
Emotion recognition from images has focused on faces or single individuals, lacking datasets that capture the overall emotional content of complex multi-person social scenes.
Approach
Annotates whole images (not individuals) with valence, arousal, and Plutchik's Wheel of Emotions labels collected via Prolific, with a public single-annotation training set and a private multi-annotation test set, and benchmarks transfer learning and late-fusion models on it.
Key results
- Baseline Emo8 classification and valence/arousal regression on FindingEmo are hard; VGG16 is the strongest CNN baseline, and ViT models (CLIP, DINOv2) are clearly superior for valence and arousal regression.
- Late fusion with facial emotion features gives the largest gains; for CLIP and DINOv2, no stream combination improved performance.
- Annotators agreed on a single Emo8 leaf for only 26.2% of multi-annotated images, and 42.8% of multi-label cases were adjacent leaves; arousal agreement is lower than valence agreement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an inverse-frequency CNN-label sampling system to select images for annotation | FindingEmo annotation collection · Public-set emotion-label distribution | annotation share: 35.9% to 28.4% (joy); 6.3% to 10.5% (sadness) | Mixed |
| Use ImageNet-pretrained rather than Places365-trained models | Emotion recognition models · FindingEmo | performance | Helped |
| Use UnbalancedCrossEntropyLoss rather than CrossEntropyLoss | Emotion classification models · FindingEmo Emo8 | performance: 9 out of 13 cases | Mixed |
| Use VGG rather than the other tested architectures | Emotion recognition models · FindingEmo | performance | Helped |
| Add facial emotion features through late fusion | Late-fusion models · FindingEmo Emo8 classification | performance | Helped |
| Add both EmoNet and OIToFER features instead of either separately | VGG16 late-fusion model · FindingEmo Emo8 classification | performance | Mixed |
| Concatenate Places365 features rather than Places365 predictions | Late-fusion models · FindingEmo Emo8 classification | performance | Mixed |
| Add any tested stream combination | CLIP and DINOv2 · FindingEmo Emo8 classification | performance | Hurt |
| Use modern ViT models rather than older CNN architectures | Emotion classification models · FindingEmo Emo8 | performance | No clear effect |
| Use ViT models rather than CNN architectures | Emotion regression models · FindingEmo Arousal and Valence | performance | Helped |
Surprising findings
- Modern ViT models (CLIP, DINOv2) do not outperform older CNNs such as VGG16 on Emo8 classification.
- Improving over the baseline with late fusion is non-trivial; except for VGG16, gains are modest.
Limitations
- Public training set has only one annotation per image, raising reliability concerns.
- Images scraped from the internet inherit its biases; representation issues were not analyzed.
- Only URLs are shared due to copyright, so links can break.
Experimental setup
Datasets: FindingEmo Places365 FER2013 Open Images
Benchmarks: FindingEmo private test set
Metrics: Weighted F1 Average Precision Mean Absolute Error Spearman R
Baselines: AlexNet VGG16 ResNet18 ResNet50 ResNet101 DenseNet161 EmoNet CLIP ViT-B/32 DINOv2 ViT-B/14
Builds on: EMOTIC Plutchik's Wheel of Emotions Places365 EmoNet
Closest papers by meaning
- EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition (NeurIPS 2025 · similarity 0.86)
- OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition (ICML 2025 · similarity 0.85)
- Bridge Then Begin Anew: Generating Target-Relevant Intermediate Model for Source-Free Visual Emotion Adaptation (AAAI 2025 · similarity 0.85)
- $E^3$: Exploring Embodied Emotion Through A Large-Scale Egocentric Video Dataset (NeurIPS 2024 · similarity 0.85)
- EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessment (ICML 2026 · similarity 0.85)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.