Research map / Vision-language and multimodal / Human and embodied modeling
EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EmoNet-Face, a synthetic, expert-annotated facial emotion benchmark with a 40-category taxonomy and three datasets, plus EmpathicInsight-Face SigLIP2-based regressors that match human-expert agreement and beat VLMs on it.
Problem
Existing facial emotion recognition benchmarks cover narrow emotion spectrums, use uncontrolled imagery with occlusions, lack demographic diversity, and have limited expert annotation quality.
Approach
Generate demographically balanced synthetic full-face images across a 40-category fine-grained emotion taxonomy derived from the Handbook of Emotions, annotate them with psychology experts (continuous 0-7 ratings and binary labels) plus Gemini-generated labels for pretraining, and train SigLIP2-based MLP regressors on top.
Key results
- EmpathicInsight-Face Large reaches average pairwise weighted kappa comparable to human annotators (difference 0.019, p=0.103) and significantly outperforms proprietary and VLM annotators (p<0.001).
- Average inter-annotator weighted kappa among human experts on EmoNet-Face HQ is 0.20 and Krippendorff's alpha across 40 emotions is 0.19.
- Only two of nine evaluated VLMs produced parseable outputs under both zero-shot and multi-shot prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a specialized SIGLIP2 model on EmoNet-Face Big and fine-tune it on EmoNet-Face Binary | EmpathicInsight-Face · EmoNet-Face HQ | weighted kappa agreement with humans: Δ=0.019 | No clear effect |
| Use specialized EmpathicInsight-Face models instead of proprietary models, zero-shot or multi-shot VLMs, or random baselines | EmoNet-Face HQ emotion recognition · EmoNet-Face HQ | weighted kappa agreement with humans | Helped |
| Use multi-shot rather than zero-shot prompting | General-purpose VLMs on EmoNet-Face HQ · EmoNet-Face HQ | weighted kappa agreement with humans | No clear effect |
| Add a target-emotion hint to Gemini Flash 2.0 annotation prompts for targeted images | EmoNet-Face Big annotations · EmoNet-Face Big | images assigned a non-zero label for the hinted emotion: approximately 50% | Helped |
Surprising findings
- Most general-purpose VLMs performed no better than random guessing on agreement with human annotators.
- Binary annotation Krippendorff's alpha overall is only 0.09, showing low agreement despite high per-batch consensus.
Limitations
- Analyses restricted to static facial expressions without context or multimodal cues.
- The 40-category taxonomy is one perspective and its cross-cultural universality is untested.
- Synthetic images may not generalize to naturalistic real-world images.
Experimental setup
Datasets: EmoNet-Face Big EmoNet-Face Binary EmoNet-Face HQ
Benchmarks: EmoNet-Face HQ
Metrics: weighted kappa Krippendorff's alpha Spearman's rho Mann-Whitney U
Baselines: Gemini 2.5 Pro Gemini 2.5 Flash Gemini 2.0 Flash Claude 3.7 Sonnet GPT-4o Nova Pro Pixtral Large Qwen Plus HumeFace Random Baseline
Builds on: SigLIP2 Theory of Constructed Emotion Handbook of Emotions
Closest papers by meaning
- Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis (AAAI 2026 · similarity 0.88)
- $E^3$: Exploring Embodied Emotion Through A Large-Scale Egocentric Video Dataset (NeurIPS 2024 · similarity 0.88)
- EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessment (ICML 2026 · similarity 0.87)
- EmoEdit: Evoking Emotions through Image Manipulation (CVPR 2025 · similarity 0.87)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.87)
- Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.