Research map / 3D vision / Face and shape
3D Facial Expressions through Analysis-by-Neural-Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SMIRK, a 3D face reconstruction method that replaces differentiable rendering with a U-Net neural renderer for self-supervised training and adds an augmented expression cycle-consistency loss to better capture subtle and extreme expressions.
Problem
Monocular 3D face reconstruction from in-the-wild images tends to miss subtle, extreme, asymmetric, or rare facial expressions, partly due to shortcomings of differentiable-rendering-based self-supervised losses and limited expression diversity in training data.
Approach
Replace the differentiable graphics renderer with a U-Net neural renderer that takes a monochrome FLAME rendering plus sparsely sampled input pixels to reconstruct the image, so supervision focuses on geometry; and generate new expression-augmented image-parameter pairs via the neural renderer with a cycle-consistency loss on expression parameters.
Key results
- On AffectNet emotion recognition, SMIRK reaches E-ACC 0.64 and V-CCC 0.72, comparable to Deep3DFace (0.65) and behind EMOCA v1 (0.68), outperforming DECA (0.59) and FOCUS (0.58).
- In a user study with 85 participants, SMIRK was preferred over DECA 603 to 77, EMOCAv2 461 to 219, Deep3D 510 to 170, and FOCUS 534 to 146, all significant at p<0.01.
- Image reconstruction by a U-Net trained on rendered geometry on AffectNet gives VGG loss 0.76 for SMIRK versus 0.78 for Deep3DFace and FOCUS, and 0.80 for DECA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove the emotion loss (ablation) | SMIRK · AffectNet test | E-ACC: 0.64 -> 0.62 | Hurt |
| remove the emotion loss (ablation) | SMIRK · AffectNet test | V-CCC: 0.72 -> 0.71 | Hurt |
| remove the emotion loss (ablation) | SMIRK · AffectNet test | A-RMSE: 0.31 -> 0.32 | Hurt |
| omit landmark loss throughout training (ablation) | SMIRK · AffectNet test | L1 loss: 0.093 -> 0.111 | Hurt |
| omit landmark loss throughout training (ablation) | SMIRK · AffectNet test | VGG loss: 0.714 -> 0.757 | Hurt |
| omit landmark loss throughout training (ablation) | SMIRK · AffectNet test | Cycle loss: 0.544 -> 0.588 | Hurt |
| use landmark loss for a few epochs, then remove it (ablation) | SMIRK · AffectNet test | L1 loss: 0.093 -> 0.093 | No clear effect |
| use landmark loss for a few epochs, then remove it (ablation) | SMIRK · AffectNet test | VGG loss: 0.714 -> 0.713 | No clear effect |
| use landmark loss for a few epochs, then remove it (ablation) | SMIRK · AffectNet test | Cycle loss: 0.544 -> 0.487 | Helped |
| add the augmented expression cycle path (ablation) | SMIRK · facial expression reconstruction | mouth-closure artifact reduction | Helped |
| add cycle-path expression augmentations (ablation) | SMIRK · facial expression reconstruction | expression detail | Helped |
Surprising findings
- The landmark loss acts mainly as a training regularizer: removing it after a few epochs gives performance close to full training.
- EMOCA v1 achieves higher emotion accuracy but often overexaggerates expressions, so emotion recognition scores do not fully reflect reconstruction faithfulness.
Limitations
- Sensitive to occlusions, since training data contains none, and tends to exaggerate expressions when parts are missing.
- Trained on single images only; temporal aspects are not explored.
- Does not estimate albedo or lighting, which limits applications such as 3D facial animation and video editing.
Experimental setup
Datasets: FFHQ CelebA LRS3 MEAD FaMoS AffectNet
Benchmarks: AffectNet test set MEAD test set
Metrics: V-CCC V-RMSE A-CCC A-RMSE E-ACC L1 loss VGG loss cycle loss user preference
Baselines: DECA EMOCA v1 EMOCA v2 Deep3DFace FOCUS MGCNet 3DDFA-v2
Builds on: DECA EMOCA FLAME MICA pix2pix-style U-Net translator
Closest papers by meaning
- TEASER: Token Enhanced Spatial Modeling for Expressions Reconstruction (ICLR 2025 · similarity 0.90)
- Learning Complete 3D Morphable Face Models From Images and Videos (CVPR 2021 · similarity 0.90)
- High-Fidelity 3D Head Avatars Reconstruction through Spatially-Varying Expression Conditioned Neural Radiance Field (AAAI 2024 · similarity 0.89)
- OFER: Occluded Face Expression Reconstruction (CVPR 2025 · similarity 0.88)
- SFDM: Robust Decomposition of Geometry and Reflectance for Realistic Face Rendering from Sparse-view Images (CVPR 2025 · similarity 0.86)
- ID-to-3D: Expressive ID-guided 3D Heads via Score Distillation Sampling (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.