Research map / Label-efficient and robust learning / Recognition and long-tail
To Err Like Human: Affective Bias-Inspired Measures for Visual Emotion Recognition Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ECC and EMC, distance-weighted accuracy-style metrics for visual emotion recognition built on Mikel's emotion wheel, and applies them to semi-supervised pseudo-labeling and model selection with a user study.
Problem
Standard accuracy treats all misclassifications equally, ignoring that confusing emotions that are psychologically similar is less severe than confusing dissimilar ones; no metric captures misclassification severity for visual emotion recognition.
Approach
Define an emotional distance from Mikel's wheel (wheel steps, with a penalty constant C for crossing polarity) and use its reciprocal to weight confusion-matrix entries, yielding ECC (overall) and EMC (misclassification-only) measures; also use EMC to adapt FixMatch-style pseudo-label thresholds and ListMLE-based ordering loss to favor better-ranked models.
Key results
- In semi-supervised learning on FI and EmoSet, the EMC-adjusted threshold method reaches 31.2% (FI, 80 labels) versus 28.2% for FixMatch and 29.7% for FlexMatch, and the method outperforms S2-VER by about 1% at most label settings (e.g., 61.5% vs 60.2% on EmoSet with 8000 labels).
- In a user study with 30 participants over 30 images (900 votes), 487 votes favored the model with higher ECC versus 242 for higher ACC, with users preferring the higher-ECC model on 24 of 30 images (80%).
- On FI with ResNet101, the combined loss (alpha=0.2) variant gives ACC 67.7 and ECC 77.8, versus 67.9 and 77.6 for cross-entropy only, while EMC rises from 52.2 to 55.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EMC-based threshold adjustment instead of FlexMatch or FixMatch thresholding | Semi-supervised visual emotion recognition · FI and EmoSet | ACC: 28.2±0.78 -> 31.2±0.12; 37.4±0.51 -> 40.8±0.34; 42.2±0.29 -> 42.7±0.21; 31.1±0.41 -> 31.6±0.56; 42.3±0.65 -> 43.7±0.69; | Helped |
| Select pseudo-labeling models using ECC/EMC and train with the combined loss | S2-VER semi-supervised emotion recognition · FI and EmoSet | ACC: 39.1±0.66 -> 40.2±1.08; 46.9±0.46 -> 48.9±0.91; 51.8±0.21 -> 52.1±0.33; 44.9±0.35 -> 47.0±0.18; 57.5±0.51 -> 59.0±0.33; | Helped |
| Replace cross-entropy with the combined loss Lc (α=1) (ablation) | ResNet50 emotion classifier · FI | ACC: 67.3 -> 65.7 | Hurt |
| Replace cross-entropy with the combined loss Lc (α=1) (ablation) | ResNet50 emotion classifier · FI | EMC: 51.5 -> 57.3 | Helped |
| Replace cross-entropy with the combined loss Lc (α=1) (ablation) | ResNet50 emotion classifier · EmoSet | ACC: 76.3 -> 74.5 | Hurt |
| Replace cross-entropy with the combined loss Lc (α=1) (ablation) | ResNet50 emotion classifier · EmoSet | EMC: 58.5 -> 61.7 | Helped |
| Use the proposed emotional label rank instead of reverse or random scrambled label ranks (ablation) | ListMLE-trained visual emotion classifier · FI | EMC: 43.3 -> 51.5; 48.6 -> 51.5 | Helped |
| Use the proposed emotional label rank instead of reverse or random scrambled label ranks (ablation) | ListMLE-trained visual emotion classifier · FI | ACC: 63.4 -> 65.7; 63.8 -> 65.7 | Helped |
Surprising findings
- Models with higher accuracy can have lower ECC/EMC; the ListMLE-combined loss reduces ACC while improving ACC2 and EMC.
- Adding an order-based loss to reduce misclassification severity reduces accuracy, a trade-off also noted in prior hierarchy-aware work.
Limitations
- Evaluated only on single-label image emotion datasets with Mikel's eight-category wheel; generalization to other emotion models is argued but not tested.
- Threshold adjustment is a simple method; the authors note more sophisticated uses of the measures are unexplored.
- Application to pretrained models and large language models is left unexplored.
Experimental setup
Datasets: FI EmoSet UnbiasedEmo
Benchmarks: FI EmoSet
Metrics: ACC ECC EMC ACC2
Baselines: FixMatch FlexMatch CoMatch SimMatch FreeMatch SoftMatch S2-VER ACC2
Builds on: FixMatch FlexMatch S2-VER ListMLE Mikel's emotion wheel confusion entropy
Closest papers by meaning
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.86)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.85)
- Bridge Then Begin Anew: Generating Target-Relevant Intermediate Model for Source-Free Visual Emotion Adaptation (AAAI 2025 · similarity 0.85)
- EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition (NeurIPS 2025 · similarity 0.84)
- OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition (ICML 2025 · similarity 0.84)
- AffectGPT-RL: Revealing Roles of Reinforcement Learning in Open-Vocabulary Emotion Recognition (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.