Research map / 3D vision / Face and shape
FaceLift: Semi-supervised 3D Facial Landmark Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Semi-supervised 3D facial landmark localization that lifts hand-labeled 2D landmarks into 3D using multi-view 3D-aware GAN samples and multi-frame in-the-wild videos, with no 3D landmark labels.
Problem
3D facial landmark localization trained on 3DMM-registered datasets lacks alignment with human-labeled 2D landmark definitions (e.g., eyebrows), creating a 2D-3D definition gap and limiting accuracy.
Approach
Lift visible 2D landmarks from a SoTA 2D detector into 3D pseudo-labels via masked, occlusion-aware multi-view optimization on 3D-aware GAN samples, then train a ViT-based transformer regressor jointly on multi-view GAN samples (3D losses) and multi-frame videos (2D Laplacian log-likelihood losses), with no 3D ground truth.
Key results
- On DAD-3DHeads, Ours achieves NMLC 1.88 overall vs 2.41 for FAN3D and 2.06 for DAD-3DNet+ (about 22% improvement over prior SoTA).
- On Multiface, Ours achieves NMLC 2.52 overall vs 3.13 for FAN3D and 3.28 for DAD-3DNet+ (about 19% improvement).
- Training with only multi-frame or only multi-view samples degrades performance, showing both data sources are needed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FaceLift instead of supervised landmark baselines | 3D facial landmark localization · DAD-3DHeads | NMLC (full): 2.08 -> 1.91 | Helped |
| use FaceLift instead of supervised landmark baselines | 3D facial landmark localization · Multiface | NMLC (full): 3.13 -> 2.52 | Helped |
| replace FaRL ViT with Resnet50 (ablation) | FaceLift · DAD-3DHeads | NMLC (full): 1.91 -> 2.23 | Hurt |
| replace FaRL ViT with Resnet152 (ablation) | FaceLift · DAD-3DHeads | NMLC (full): 1.91 -> 2.06 | Hurt |
| replace FaRL ViT with Resnet50 (ablation) | FaceLift · Multiface | NMLC (full): 2.52 -> 2.87 | Hurt |
| replace FaRL ViT with Resnet152 (ablation) | FaceLift · Multiface | NMLC (full): 2.52 -> 2.62 | Hurt |
| remove multi-view GAN samples from training (ablation) | FaceLift · DAD-3DHeads | NMLC (full): 1.91 -> 2.06 | Hurt |
| remove multi-frame video samples from training (ablation) | FaceLift · DAD-3DHeads | NMLC (full): 1.91 -> 2.33 | Hurt |
| remove multi-view GAN samples from training (ablation) | FaceLift · Multiface | NMLC (full): 2.52 -> 3.51 | Hurt |
| remove multi-frame video samples from training (ablation) | FaceLift · Multiface | NMLC (full): 2.52 -> 2.98 | Hurt |
| increase training videos/GAN latents from 100 to the full training set (ablation) | FaceLift · DAD-3DHeads | NMLC (full): 2.12 -> 1.91 | Helped |
| increase training videos/GAN latents from 1k to the full training set (ablation) | FaceLift · DAD-3DHeads | NMLC (full): 2.00 -> 1.91 | Helped |
Surprising findings
- Swapping the FaRL ViT encoder for a ResNet152 of similar parameter count gave only a slight drop, while occlusion performance dropped more significantly.
Limitations
- Performance is bounded by the quality of the 2D landmark detector, which fails on puckers and asymmetric expressions.
- Depends on 3D-aware GAN quality; GANs have limited pose and expression distributions.
- Pre-processing took roughly 3 days on a single A10G GPU.
Experimental setup
Datasets: CelebV-HQ IDE-3D GAN samples (FFHQ) WFLW LaPa DAD-3DHeads Multiface
Benchmarks: DAD-3DHeads Multiface
Metrics: NMLC (normalized mean local consistency)
Baselines: SynergyNet 3DDFA 3DDFA+ 3DDFAv2 DAD-3DNet DAD-3DNet+ FAN3D
Builds on: IDE-3D FaRL STAR Loss / Laplacian log-likelihood (Kumar et al.) 3DDFA-V2 SynergyNet
Closest papers by meaning
- Improving Landmark Localization With Semi-Supervised Learning (CVPR 2018 · similarity 0.89)
- Learning to Detect 3D Facial Landmarks via Heatmap Regression with Graph Convolutional Network (AAAI 2022 · similarity 0.89)
- Continuous Landmark Detection With 3D Queries (CVPR 2023 · similarity 0.88)
- Generalizable Face Landmarking Guided by Conditional Face Warping (CVPR 2024 · similarity 0.86)
- Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation (NeurIPS 2024 · similarity 0.85)
- Semi-supervised 3D Semantic Scene Completion with 2D Vision Foundation Model Guidance (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.