Research map / Probabilistic and scientific modeling / Causal discovery
Robust Emotion Recognition in Context Debiasing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLEF, a model-agnostic counterfactual causal framework that adds a non-invasive context branch to estimate and subtract the direct context bias in context-aware emotion recognition, improving five existing CAER methods on EMOTIC and CAER-S.
Problem
Context-aware emotion recognition models learn spurious correlations between background context and emotion labels (context bias), which degrades prediction robustness on uncontrolled images.
Approach
Builds a generalized CAER causal graph and uses counterfactual inference: a context branch (ResNet-152 on subject-masked images) estimates the direct context bias, and at inference the NDE is subtracted from the total effect (TIE), with a learned uniform no-treatment embedding regularized by KL divergence.
Key results
- CLEF improves mAP on EMOTIC by 3.74 (EMOT-Net, 27.93 to 31.67), 3.59 (CAER-Net), 4.02 (GNN-CNN), 3.64 (CD-Net) and 2.77 (EmotiCon, to 38.05).
- On CAER-S, CLEF raises accuracy by 2.52, 2.39, 2.32, 3.08 and 1.97 points across the five CAER methods.
- CLEF outperforms the prior debiasing method CCIM on both datasets (e.g., EMOT-Net on EMOTIC: 31.67 vs 30.88).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CLEF to existing CAER methods | EMOT-Net, CAER-Net, GNN-CNN, CD-Net, and EmotiCon · EMOTIC | mAP (%): 27.93 -> 31.67; 23.85 -> 27.44; 28.16 -> 32.18; 28.87 -> 32.51; 35.28 -> 38.05 | Helped |
| Add CLEF to existing CAER methods | EMOT-Net, CAER-Net, GNN-CNN, CD-Net, and EmotiCon · CAER-S | accuracy (%): 74.51 -> 77.03; 73.47 -> 75.86; 77.21 -> 79.53; 85.33 -> 88.41; 88.65 -> 90.62 | Helped |
| Remove the vanilla CAER model (ablation) | CD-Net + CLEF · EMOTIC and CAER-S | mAP / accuracy: 32.51 -> 19.64; 88.41 -> 62.87 | Hurt |
| Remove the additional context branch (ablation) | CD-Net + CLEF · EMOTIC and CAER-S | mAP / accuracy: 32.51 -> 28.55; 88.41 -> 85.54 | Hurt |
| Remove KL regularization (ablation) | CD-Net + CLEF · EMOTIC and CAER-S | mAP / accuracy: 32.51 -> 32.26; 88.41 -> 88.09 | Hurt |
| Remove subject masking in the additional context branch (ablation) | CD-Net + CLEF · EMOTIC and CAER-S | mAP / accuracy: 32.51 -> 31.38; 88.41 -> 87.68 | Hurt |
| Use ImageNet rather than Places365 pre-training for the context branch (ablation) | CD-Net + CLEF · EMOTIC and CAER-S | mAP / accuracy: 32.51 -> 30.74; 88.41 -> 87.35 | Hurt |
| Use ResNet-50 instead of ResNet-152 for the context branch (ablation) | CD-Net + CLEF · EMOTIC and CAER-S | mAP / accuracy: 32.51 -> 31.45; 88.41 -> 87.83 | Hurt |
| Use VGG-16 instead of ResNet-152 for the context branch (ablation) | CD-Net + CLEF · EMOTIC and CAER-S | mAP / accuracy: 32.51 -> 29.93; 88.41 -> 86.76 | Hurt |
| Replace the uniform no-treatment embedding with average feature embedding (ablation) | CD-Net + CLEF and EmotiCon + CLEF · EMOTIC and CAER-S | mAP / accuracy: CD-Net: 28.87 -> 27.85, 85.33 -> 83.06; EmotiCon: 35.28 -> 33.18, 88.65 -> 85.67 | Hurt |
| Replace the uniform no-treatment embedding with random feature embedding (ablation) | CD-Net + CLEF and EmotiCon + CLEF · EMOTIC and CAER-S | mAP / accuracy: CD-Net: 28.87 -> 24.61, 85.33 -> 76.43; EmotiCon: 35.28 -> 28.77, 88.65 -> 78.25 | Hurt |
| Replace CLEF with CCIM | EMOT-Net and CAER-Net · EMOTIC and CAER-S | mAP / accuracy: EMOT-Net: 31.67 vs 30.88, 77.03 vs 75.82; CAER-Net: 27.44 vs 26.51, 75.86 vs 74.81 | Hurt |
Surprising findings
- Replacing the uniform no-treatment assumption with average or random feature embeddings yields results worse than the vanilla baseline.
Limitations
- Gains on CAER-S are smaller than on EMOTIC, attributed to less context richness in CAER-S.
- The no-treatment condition must be approximated by a learned uniform-distribution parameter because neural models cannot take void inputs.
Experimental setup
Datasets: EMOTIC CAER-S Places365
Benchmarks: EMOTIC CAER-S
Metrics: mAP accuracy
Baselines: EMOT-Net CAER-Net GNN-CNN CD-Net EmotiCon HLCR TEKG RRLA VRD SIB-Net MCA GRERN
Builds on: EmotiCon CCIM Causal intervention for visual dialogue (TIE)
Closest papers by meaning
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.84)
- Knowledge-Aligned Counterfactual-Enhancement Diffusion Perception for Unsupervised Cross-Domain Visual Emotion Recognition (CVPR 2025 · similarity 0.84)
- Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis (AAAI 2026 · similarity 0.84)
- Uncertain Multimodal Intention and Emotion Understanding in the Wild (CVPR 2025 · similarity 0.84)
- AffectGPT-RL: Revealing Roles of Reinforcement Learning in Open-Vocabulary Emotion Recognition (NeurIPS 2026 · similarity 0.84)
- OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.