Research map / Label-efficient and robust learning / Recognition and long-tail
Multi-scale Dynamic and Hierarchical Relationship Modeling for Facial Action Units Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MDHR, combining multi-scale temporal differencing with adaptive weighting (MFD) and a two-stage local/cross-region graph relationship module (HSR) to recognize facial action units in video, reaching SOTA F1 on BP4D and DISFA.
Problem
Facial action unit (AU) occurrence recognition struggles to model hierarchical spatio-temporal AU dependencies and the heterogeneous motion range and magnitude of different AUs.
Approach
Computes frame-difference features at multiple backbone layers, adaptively weights each spatial scale per AU (MFD), then models AU relations hierarchically: first within upper/middle/lower facial regions (AU-specific extractors plus auxiliary regional-combination loss), then across regions via a GAT over activated AUs (HSR), followed by TCN.
Key results
- MDHR with ResNet-50 achieves 66.6% average F1 on BP4D, surpassing prior best ME-GraphAU (65.5%) and WSRTL (65.9%).
- With Swin-B, MDHR reaches 66.2% average F1 on BP4D and 66.2% on DISFA.
- On DISFA, Swin-B variant reaches 65.4% vs. 64.6% for WSRTL (ResNet-50 variant 61.4%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MDHR with a ResNet-50 backbone instead of WSRTL | AU occurrence recognition · BP4D | F1-score (%): 65.9 -> 66.6 | Helped |
| Use MDHR with a Swin-B backbone instead of WSRTL | AU occurrence recognition · DISFA | F1-score (%): 64.6 -> 66.2 | Helped |
| Add multi-scale facial dynamic modeling (MFD) (ablation) | Backbone + AU-specific feature extractors on BP4D · BP4D | F1-score (%): 63.3 -> 64.6 | Helped |
| Add hierarchical spatio-temporal AU relationship modeling (HSR) (ablation) | Backbone + AU-specific feature extractors on BP4D · BP4D | F1-score (%): 63.3 -> 65.1 | Helped |
| Add AU-specific feature extractors for local AU relationships (ablation) | HSR on BP4D · BP4D | F1-score (%): 63.3 -> 64.1 | Helped |
| Add the auxiliary regional AU-combination prediction branch (ablation) | HSR on BP4D · BP4D | F1-score (%): +0.4% | Helped |
| Add cross-regional AU relationship modeling (ablation) | HSR on BP4D · BP4D | F1-score (%) | Helped |
| Combine facial dynamics from all spatial scales with adaptive weighting (ablation) | MFD on BP4D · BP4D | F1-score (%): 64.1 -> 65.3 | Helped |
| Replace adaptive weighting with summation of multi-scale dynamics (ablation) | MFD on BP4D · BP4D | F1-score (%): 65.3 -> 64.9 | Hurt |
| Replace adaptive weighting with concatenation of multi-scale dynamics (ablation) | MFD on BP4D · BP4D | F1-score (%): 65.3 -> 64.9 | Hurt |
| Use activated-AU connections across different facial regions (ablation) | Cross-regional AU relationship modeling on BP4D · BP4D | F1-score (%): 64.6 -> 65.1 | Helped |
Surprising findings
- The Swin-B backbone does not beat ResNet-50 on BP4D (66.1% vs 66.6%) despite stronger capacity.
Limitations
- Facial region slicing strategy is fixed and could be improved.
- Graph edge learning is relatively simple and could be made more advanced.
Experimental setup
Datasets: BP4D DISFA
Benchmarks: BP4D DISFA
Metrics: frame-based F1 score
Baselines: EAC-Net JAA-Net ARL SMA-Net SRERL FAUDT FAN-Trans ME-GraphAU STRAL HSTR-Net KS WSRTL
Builds on: temporal difference network ME-GraphAU asymmetric loss temporal convolution network graph attention network
Closest papers by meaning
- Semantic Relationships Guided Representation Learning for Facial Action Unit Recognition (AAAI 2019 · similarity 0.85)
- Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution (AAAI 2020 · similarity 0.84)
- Facial Action Unit Detection With Transformers (CVPR 2021 · similarity 0.84)
- Deep Region and Multi-Label Learning for Facial Action Unit Detection (CVPR 2016 · similarity 0.84)
- Knowledge-Driven Self-Supervised Representation Learning for Facial Action Unit Recognition (CVPR 2022 · similarity 0.83)
- Distribution Matching for Multi-Task Learning of Classification Tasks: A Large-Scale Study on Faces & Beyond (AAAI 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.