Research map / Vision-language and multimodal / Human and embodied modeling
OUS: Bridging Scene Context and Facial Features to Overcome the Rigid Cognitive Problem
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OUS, a dual-stream dynamic facial expression recognition model that fuses scene context with facial features via polarity, similarity, and contrastive losses and a cross-scale learnable-query encoder (MCFE).
Problem
Dynamic facial expression recognition (DFER) methods treat scene context as noise and rely only on facial cues, causing a cognitive gap with human annotators who use scene context to resolve ambiguous expressions (the "Rigid Cognitive Problem").
Approach
Treat scene video as a second stream aligned with facial features via cosine similarity loss, extract scene polarity with a polarity loss from early vision-encoder layers, and fuse both with a Multimodal Cross-Scale Fusion Encoder using shared learnable queries, plus learnable-prompt contrastive loss.
Key results
- OUS reaches 64.33 UAR / 74.02 WAR on DFEW, improving over the prior best (LSGTNet, 61.33 / 72.34) by about 1.68 points.
- On FERV39k, OUS reaches 42.43 UAR / 53.30 WAR, a 1.65-point gain over the prior state of the art (DFER-CLIP, 41.27 / 51.65 per Table 2 comparisons).
- Removing multi-loss constraints and MCFE lowers DFEW UAR from 64.33 to 55.12 for cross-entropy only; the multi-loss plus MCFE configuration converges in 5 epochs vs 10 for cross-entropy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use OUS instead of LSGTNet | DFEW · DFEW | UAR: 61.33 -> 64.33 | Helped |
| use OUS instead of LSGTNet | DFEW · DFEW | WAR: 72.34 -> 74.02 | Helped |
| use OUS instead of LSGTNet | FERV39k · FERV39k | UAR: 41.30 -> 42.43 | Helped |
| use OUS instead of LSGTNet | FERV39k · FERV39k | WAR: 51.31 -> 53.30 | Helped |
| add multi-loss constraints (similarity, polarity, and contrastive losses) to cross-entropy (ablation) | OUS · FERV39k | UAR: 36.65 -> 38.48 | Helped |
| add multi-loss constraints (similarity, polarity, and contrastive losses) to cross-entropy (ablation) | OUS · DFEW | UAR: 55.12 -> 59.12 | Helped |
| add MCFE to the multi-loss OUS configuration (ablation) | OUS · FERV39k | UAR: 38.48 -> 42.43 | Helped |
| add MCFE to the multi-loss OUS configuration (ablation) | OUS · DFEW | UAR: 59.12 -> 64.33 | Helped |
| increase prompt length from 16 to 64 with 12 MCFE blocks (ablation) | OUS · DFEW | UAR: 0.5842 -> 0.6433 | Helped |
| increase prompt length from 16 to 64 with 12 MCFE blocks (ablation) | OUS · DFEW | WAR: 0.7188 -> 0.7402 | Helped |
| vary MCFE block count from 8 to 12 to 16 at prompt length 64 (ablation) | OUS · DFEW | UAR: 0.6097; 0.6433; 0.6162 | Mixed |
| vary MCFE block count from 8 to 12 to 16 at prompt length 64 (ablation) | OUS · DFEW | WAR: 0.7368; 0.7402; 0.7402 | Mixed |
Surprising findings
- Scene context is treated as a useful signal rather than noise, and the authors report gains on classes like happiness, sadness, and disgust that are often confused from faces alone.
Limitations
- Scene-polarity and facial-recognition preprocessing details are sparse, and the facial recognition module is only described at a high level.
- Performance on Surprise and Fear on DFEW is lower than some baselines (e.g., Surprise 60.98 vs 65.01 for LSGTNet), indicating uneven per-class gains.
Experimental setup
Datasets: DFEW FERV39k
Benchmarks: DFEW FERV39k
Metrics: UAR WAR
Baselines: C3D I3D-RGB P3D 3D ResNet18 R(2+1)D18 Former-DFER NR-DFERNet SW-FSCL LSGTNet CLIPER DFER-CLIP EmoCLIP
Builds on: CLIP CoOp Vision Transformer
Closest papers by meaning
- Lifting Scheme-Based Implicit Disentanglement of Emotion-Related Facial Dynamics in the Wild (AAAI 2025 · similarity 0.88)
- D2SP: Dynamic Dual-Stage Purification Framework for Dual Noise Mitigation in Vision-based Affective Recognition. (CVPR 2025 · similarity 0.87)
- Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis (AAAI 2026 · similarity 0.86)
- EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition (NeurIPS 2025 · similarity 0.86)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.84)
- EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessment (ICML 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.