Research map / 3D vision / Face and shape
Learning Person-Specific Animatable Face Models from In-the-Wild Images via a Shared Base Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a self-supervised ViT-MAE base model for animatable 3D face reconstruction from in-the-wild images, then transfers it to person-specific models via lightweight adapters, using a silhouette vertex re-projection loss and a UNet-teacher-student loss for details.
Problem
Generic 3D face reconstruction models trained on in-the-wild images underfit individual-specific shape and detail, and existing landmark losses suffer from boundary "landmark marching" under pose variation; animatable wrinkle details also remain hard to learn without 3D data.
Approach
Build a scalable ViT-MAE base model trained self-supervised, add a silhouette vertex re-projection loss that matches silhouette landmarks to 3D edge vertices chosen by normal sign, supervise a MAE detail network with a UNet teacher's displacement maps, and personalize by training only adapters (plus detail latent and LayerNorm) on a person's images or video.
Key results
- On 300-W, Ours-base reaches boundary 0.0559 and inner 0.0348 landmark error versus DECA's 0.0576 and 0.0402.
- On FaceScape, the person-specific model reaches inner error 0.0253 and overall 0.0305, versus DECA's 0.0477 and 0.0496.
- Ours-base w/o silhouette loss improves boundary and inner landmark errors by 9.1% and 13.2% when Lsil is added, per the paper's ablation on 300-W.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add silhouette vertex re-projection loss (ablation) | Ours-base · 300-W | boundary RMSE error: 0.0616 -> 0.0559 | Helped |
| add silhouette vertex re-projection loss (ablation) | Ours-base · 300-W | inner RMSE error: 0.0401 -> 0.0348 | Helped |
| add silhouette vertex re-projection loss (ablation) | Ours-base · 300-W | overall RMSE error: 0.0455 -> 0.0401 | Helped |
| add silhouette vertex re-projection loss (ablation) | Ours-base · 300-VW | overall RMSE error: 0.0580 -> 0.0488 | Helped |
| add silhouette vertex re-projection loss (ablation) | Ours-base · FaceScape | overall error: 0.0539 (±0.0181) -> 0.0429 (±0.0128) | Helped |
| add person-specific adapters | Ours-base · 300-VW | overall RMSE error: 0.0488 -> 0.0252 | Helped |
| add person-specific adapters | Ours-base · FaceScape | overall error: 0.0429 (±0.0128) -> 0.0305 (±0.0060) | Helped |
| remove teacher-student supervision loss (ablation) | Detail MAE · Facial detail reconstruction | detail realism | Hurt |
| replace the MAE with convolutional and deconvolutional networks (ablation) | Detail reconstruction network · Facial detail reconstruction | expression-dependent detail capture | Hurt |
Surprising findings
- The base model without the silhouette loss performs slightly worse than DECA on 300-W overall error, which the authors attribute to different training data.
Limitations
- Person-specific transfer needs multiple images or a video of the individual.
- Occluded regions are not optimized during transfer and rely on base-model priors.
- 300-VW annotations are semi-automatic, so that benchmark serves only as a reference.
Experimental setup
Datasets: BUPT-Balancedface Celeb-DF (v2) MEAD 300-W 300-VW FaceScape FaceForensics++
Benchmarks: 300-W 300-VW FaceScape
Metrics: RMSE boundary landmark error RMSE inner landmark error overall landmark error
Baselines: DECA EMOCA-v2 SynergyNet 3DDFA-v2 FaceScape FaceVerse
Builds on: DECA FLAME EMOCA ViT MAE Adapters
Closest papers by meaning
- Learning Complete 3D Morphable Face Models From Images and Videos (CVPR 2021 · similarity 0.88)
- Self-Supervised Multi-Level Face Model Learning for Monocular Reconstruction at Over 250 Hz (CVPR 2018 · similarity 0.88)
- Low-Rank Head Avatar Personalization with Registers (NeurIPS 2025 · similarity 0.86)
- Generalizable Face Landmarking Guided by Conditional Face Warping (CVPR 2024 · similarity 0.86)
- FRESA: Feedforward Reconstruction of Personalized Skinned Avatars from Few Images (CVPR 2025 · similarity 0.85)
- Visual Persona: Foundation Model for Full-Body Human Customization (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.