Research map / Label-efficient and robust learning / Recognition and long-tail
AUEditNet: Dual-Branch Facial Action Unit Intensity Manipulation with Implicit Disentanglement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AUEditNet, a dual-branch StyleGAN2 W+ latent editor that manipulates intensities of 12 facial action units using only 18 DISFA training subjects, without extra loss terms or large batches.
Problem
AU intensity manipulation in face images is hard to train because public datasets with multi-AU intensity labels have very few subjects, causing identity and attribute entanglement.
Approach
A dual-branch design where the target branch is fed a randomly chosen image from another subject, so that only target-attribute information fills the editing embedding space, plus a level-wise label latent space (label encoder/decoder) that maps AU labels to per-W+-level pseudo-labels, removing the need for batch-level normalization losses.
Key results
- On DISFA, AUEditNet reaches average ICC 0.628 and MSE 0.283 for AU intensity editing, versus 0.617 and 0.290 for ReDirTrans (average over 12 AUs).
- When deactivating all AUs, AUEditNet achieves a 38.92% MSE improvement over ReDirTrans.
- Identity preservation improves 7.33% (real target) and 9.19% (inverted target) over ReDirTrans, though DeltaEdit preserves identity best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AUEditNet instead of ReDirTrans for AU intensity editing | AU intensity editing · DISFA test | ICC(3,1) average: .617 -> .628 | Helped |
| Use AUEditNet instead of ReDirTrans for AU intensity editing | AU intensity editing · DISFA test | MSE average: .290 -> .283 | Helped |
| Use AUEditNet instead of ReDirTrans for source AU removal | Neutral face generation · DISFA test | MSE average: .167 -> .102 | Helped |
| Use AUEditNet instead of DeltaEdit | Facial attribute editing · DISFA test, real target | identity distance: .396 -> .468 | Hurt |
| Use AUEditNet instead of DeltaEdit | Facial attribute editing · DISFA test, inverted target | identity distance: .309 -> .435 | Hurt |
| Use AUEditNet instead of DeltaEdit | Facial attribute editing · DISFA test, real target | L2: .022 -> .026 | Hurt |
| Use AUEditNet instead of DeltaEdit | Facial attribute editing · DISFA test, inverted target | LPIPS: .074 -> .126 | Hurt |
| Use AUEditNet instead of the strongest reported baseline | Smile intensity manipulation · FFHQ test | ρ: 96.9 -> 121.3 | Helped |
| Remove pixel-wise and perceptual losses (ablation) | AUEditNet · DISFA test, target removal and addition | MSE: 0.288 -> 0.388 | Hurt |
| Remove pretrained function loss (ablation) | AUEditNet · DISFA test, target removal and addition | MSE: 0.288 -> 0.507 | Hurt |
| Remove identity loss (ablation) | AUEditNet · DISFA test, target removal and addition | identity distance: 0.468 -> 0.533 | Hurt |
| Remove identity loss (ablation) | AUEditNet · DISFA test, target removal and addition | ICC: 0.619 -> 0.619 | Helped |
Surprising findings
- Removing the pixel/perceptual reconstruction losses (w/o LR & LP) and the identity loss improved AU manipulation accuracy, with a well-trained AU estimator loss mattering more than paired target images.
- Despite training labels having six intensity levels, the model produces reasonable results for negative and float intensities.
Limitations
- Relies on only 18 training subjects from DISFA; generalization to out-of-domain faces is shown only qualitatively on CelebA-HQ and FFHQ.
- Identity preservation is weaker than DeltaEdit, and AU manipulation accuracy trades off against identity loss.
Experimental setup
Datasets: DISFA BU-4DFE CelebA-HQ FFHQ
Benchmarks: DISFA BU-4DFE CelebA-HQ FFHQ
Metrics: ICC(3,1) MSE identity distance L2 LPIPS Ed rho
Baselines: ReDirTrans DeltaEdit GANimation HR Aps MAE-Face Talk-to-Edit StyleFlow Do et al.
Builds on: StyleGAN2 e4e ReDirTrans DeltaEdit ArcFace
Closest papers by meaning
- Joint Representation and Estimator Learning for Facial Action Unit Intensity Estimation (CVPR 2019 · similarity 0.91)
- Weakly-Supervised Deep Convolutional Neural Network Learning for Facial Action Unit Intensity Estimation (CVPR 2018 · similarity 0.89)
- Trend-Aware Supervision: On Learning Invariance for Semi-supervised Facial Action Unit Intensity Estimation (AAAI 2024 · similarity 0.88)
- FG-EmoTalk: Talking Head Video Generation with Fine-Grained Controllable Facial Expressions (AAAI 2024 · similarity 0.85)
- Learning Adaptive Spatial Coherent Correlations for Speech-Preserving Facial Expression Manipulation (CVPR 2024 · similarity 0.85)
- Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis (AAAI 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.