Research map / Vision-language and multimodal / Human and embodied modeling
CocoER: Aligning Multi-Level Feature by Competition and Coordination for Emotion Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CocoER, a multi-level (head, body, context) emotion recognition framework that uses cross-level attention, CLIP-prototype pseudo labels, and an inference-time competition-and-coordination update of level-wise linear mappings.
Problem
Visual emotion recognition from images with multiple cropped levels (head, body, context) suffers from inconsistent per-level predictions that conflict and degrade the final emotion decision.
Approach
Uses a vocabulary-informed CLIP prototype pseudo label to identify the most inconsistent image level, eliminates such levels via a min-max (MGDA-style QP) competition process, and coordinates by updating the shared linear mappings at inference time to refine multi-level features.
Key results
- On EMOTIC, the method reaches 37.32% mAP, versus 35.16% for Hoang et al. 2021 and 35.48% for Mittal et al. 2020.
- On CAER-S, the method reaches 91.86% accuracy, versus 90.83% for Wu et al. 2024 and 84.42% for Li et al. 2021.
- Inserting the eliminated-level signal into VLM prompts or inputs raises GPT-4o mAP on EMOTIC from 23.80 to 26.64 (multi-level variant, +2.84).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CocoER instead of the strongest reported baseline | EMOTIC emotion recognition · EMOTIC test | mAP: 35.48 -> 37.32 | Helped |
| use CocoER instead of the strongest reported baseline | EMOTIC emotion recognition · EMOTIC test | C-F1 (%): 34.24 -> 36.77 | Helped |
| use CocoER instead of the strongest reported baseline | EMOTIC emotion recognition · EMOTIC test | O-F1 (%): 54.78 -> 57.23 | Helped |
| use CocoER instead of the strongest reported baseline | CAER-S emotion recognition · CAER-S test | accuracy (%): 90.83 -> 91.86 | Helped |
| replace ResNet-50 HFE with cross-level attention (ablation) | CocoER · EMOTIC | mAP: 36.71 -> 37.32 | Helped |
| replace ResNet-50 HFE with cross-level attention (ablation) | CocoER · CAER-S | accuracy: 90.16 -> 91.86 | Helped |
| replace CLIP predictions with prototype-aligned pseudo labels (ablation) | CocoER · EMOTIC | mAP: 36.12 -> 37.32 | Helped |
| replace CLIP predictions with prototype-aligned pseudo labels (ablation) | CocoER · CAER-S | accuracy: 88.28 -> 91.86 | Helped |
| add workspace competition and coordination feature refinement (ablation) | CocoER · EMOTIC | mAP: 35.73 -> 37.32 | Helped |
| add workspace competition and coordination feature refinement (ablation) | CocoER · CAER-S | accuracy: 87.60 -> 91.86 | Helped |
Surprising findings
- Zero-shot VLMs (BLIP2, LLaVA, GPT-4o) perform far below specialized emotion recognizers on CAER-S (14-29% accuracy vs. 85-91%).
Limitations
- Relies on the CLIP-based pseudo label; the paper notes the regularization term is needed to avoid over-dependence on the pseudo label.
- Inference-stage parameter updates add per-sample computation (QP solve for competition), which is not quantified in the main text.
Experimental setup
Datasets: EMOTIC CAER-S
Benchmarks: EMOTIC CAER-S
Metrics: mAP C-F1 O-F1 accuracy
Baselines: Li et al. 2021 Hoang et al. 2021 Mittal et al. 2020 Wu et al. 2024 BLIP2-6.7b LLaVa-1.6-7b GPT-4o Kosti et al. 2019 EfficientFace MA-Net CLIP-ViT-L
Builds on: CLIP MGDA Workspace Theory Grad-CAM
Closest papers by meaning
- Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis (AAAI 2026 · similarity 0.86)
- Distribution Matching for Multi-Task Learning of Classification Tasks: A Large-Scale Study on Faces & Beyond (AAAI 2024 · similarity 0.86)
- Knowledge-Aligned Counterfactual-Enhancement Diffusion Perception for Unsupervised Cross-Domain Visual Emotion Recognition (CVPR 2025 · similarity 0.84)
- CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognition (AAAI 2024 · similarity 0.84)
- Bridge Then Begin Anew: Generating Target-Relevant Intermediate Model for Source-Free Visual Emotion Adaptation (AAAI 2025 · similarity 0.83)
- FindingEmo: An Image Dataset for Emotion Recognition in the Wild (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.