Research map / Label-efficient and robust learning / Semi-supervised learning
Trend-Aware Supervision: On Learning Invariance for Semi-supervised Facial Action Unit Intensity Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Trend-Aware Supervision (TAS), three auxiliary losses (intra-trend ranking, intra-trend speed via mixup, inter-trend subject alignment) that let a ResNet34 AU intensity estimator learn invariant AU features from keyframe-only annotations.
Problem
Semi-supervised facial AU intensity estimation with only keyframe annotations suffers from spurious correlations caused by AU co-occurrence and subject variation, yielding non-robust, entangled intensity predictions.
Approach
Exploit the monotonic trends between adjacent keyframes as extra supervision on AU features (not on intensity outputs): enforce monotonic growth of feature change amplitude relative to the first frame, match feature speed to mixup-interpolated intensities within a segment, and align same-intensity keyframe features across subjects.
Key results
- On BP4D with ~2% keyframe annotations, the method reaches ICC 0.71 and MAE 0.69, the best ICC and MAE among semi-supervised baselines (RandCon ICC 0.68, MAE 0.80).
- On DISFA with ~1% keyframe annotations, the method reaches ICC 0.52 and MAE 0.36, outperforming RandCon (ICC 0.41, MAE 0.38) among semi-supervised methods.
- Ablation shows each loss helps: baseline A0 ICC 0.59 on BP4D rises to 0.71 with all three losses (A6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add intra-trend ranking awareness to separate spatial attention layers (ablation) | TAS model · BP4D | MAE: .84 -> .78 | Helped |
| Add intra-trend ranking awareness to separate spatial attention layers (ablation) | TAS model · DISFA | MAE: .63 -> .60 | Helped |
| Add intra-trend speed awareness to separate spatial attention layers (ablation) | TAS model · BP4D | MAE: .84 -> .75 | Helped |
| Add intra-trend speed awareness to separate spatial attention layers (ablation) | TAS model · DISFA | MAE: .63 -> .57 | Helped |
| Add inter-trend subject awareness to separate spatial attention layers (ablation) | TAS model · BP4D | MAE: .84 -> .79 | Helped |
| Add inter-trend subject awareness to separate spatial attention layers (ablation) | TAS model · DISFA | MAE: .63 -> .55 | Helped |
| Add all three trend-awareness losses (ablation) | TAS model · BP4D | ICC: .68 -> .71 | Helped |
| Add all three trend-awareness losses (ablation) | TAS model · DISFA | ICC: .44 -> .52 | Helped |
| Apply trend-aware supervision in S1 (segments, all frames, multi-AU labels) (ablation) | S1 baseline · BP4D | ICC: .65 -> .68 | Helped |
| Apply trend-aware supervision in S1 (segments, all frames, multi-AU labels) (ablation) | S1 baseline · DISFA | ICC: .49 -> .52 | Helped |
| Apply trend-aware supervision in S2 (segments, randomly selected 2 frames, single-AU label) (ablation) | S2 baseline · BP4D | MAE: .80 -> .73 | Helped |
| Apply subject awareness in S3 (randomly selected frames, multi-AU labels) (ablation) | S3 baseline · DISFA | ICC: .44 -> .47 | Helped |
Surprising findings
- Under the keyframe setting, the method's ICC on BP4D (0.71) is comparable to some fully supervised methods, though its MAE (0.69) is worse than SCC (0.58) and RE-Net (0.65).
Limitations
- Spurious correlation is only addressed through auxiliary losses; no formal invariance guarantee is provided.
- Performance is not uniformly best: fully supervised SCC has better MAE on BP4D, and some AUs (e.g., AU15/AU20 on DISFA) remain weak.
Experimental setup
Datasets: BP4D DISFA
Benchmarks: BP4D DISFA
Metrics: ICC(3,1) MAE
Baselines: Ladder OSVR BORMIR KJRE KBSS CFLF RandCon HBN Heatmap 2DC CCNN-IT CNN
Builds on: mixup RandCon CFLF KBSS
Closest papers by meaning
- Joint Representation and Estimator Learning for Facial Action Unit Intensity Estimation (CVPR 2019 · similarity 0.90)
- Bilateral Ordinal Relevance Multi-Instance Regression for Facial Action Unit Intensity Estimation (CVPR 2018 · similarity 0.89)
- Weakly-Supervised Deep Convolutional Neural Network Learning for Facial Action Unit Intensity Estimation (CVPR 2018 · similarity 0.89)
- AUEditNet: Dual-Branch Facial Action Unit Intensity Manipulation with Implicit Disentanglement (CVPR 2024 · similarity 0.88)
- Learning Adaptive Spatial Coherent Correlations for Speech-Preserving Facial Expression Manipulation (CVPR 2024 · similarity 0.83)
- PrefAce: Face-Centric Pretraining with Self-Structure Aware Distillation (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.