Research map / 3D vision / Face and shape
Makeup Prior Models for 3D Facial Makeup Estimation and Applications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds PCA-based and StyleGAN2-based 3D facial makeup prior models, with a regression encoder and makeup consistency loss, for fast and robust makeup estimation on 3DMM (FLAME) faces and applications like reconstruction, editing, transfer, and interpolation.
Problem
Estimating 3D facial makeup layers from single images is slow and unreliable, especially on self-occluded faces, and prior makeup residual models (Yang-Res) are limited to reconstruction and unsuitable for transfer or interpolation.
Approach
Extends 3D face priors with makeup-specific priors: a linear PCA model over makeup UV textures (RGB bases plus alpha matte) and a StyleGAN2 W+ generative prior, each paired with a regression encoder trained self-supervised through a makeup consistency module that re-estimates makeup on transferred bare faces.
Key results
- Ours (PCA) estimates makeup in 0.31s versus 56.12s for Yang-Res and 63.13s for Yang-Ext, a roughly 180-fold speedup.
- Ours (StyleGAN2) reaches 18.13s total inference (1.66s network plus 16.47s refinement), about 3x faster than Yang-Res.
- On Wild, Ours (StyleGAN2) achieves RMSE 0.0517, SSIM 0.6240, LPIPS 0.0608, outperforming Yang-Res (RMSE 0.0543, SSIM 0.6036, LPIPS 0.0674) in the DECA-based reconstruction table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the StyleGAN2 makeup prior to DECA | DECA · Wild | RMSE: 0.0876 -> 0.0807 | Helped |
| add the StyleGAN2 makeup prior to DECA | DECA · BeautyFace | RMSE: 0.0636 -> 0.0568 | Helped |
| remove StyleGAN2 coefficient refinement (ablation) | Ours (StyleGAN2) · Wild | RMSE: 0.0618 -> 0.0517 | Hurt |
| remove StyleGAN2 coefficient refinement (ablation) | Ours (StyleGAN2) · BeautyFace | RMSE: 0.0695 -> 0.0650 | Hurt |
| add the makeup consistency module and its cycle loss (ablation) | makeup estimation network · 3D facial makeup estimation | robustness | Helped |
| refine PCA makeup coefficients (ablation) | Ours (PCA) · 3D facial makeup estimation | makeup accuracy | Hurt |
| add coefficient refinement (ablation) | Ours (StyleGAN2) · 3D facial makeup estimation | makeup accuracy | Helped |
| remove the soft symmetry loss Lsym (ablation) | StyleGAN2 refinement · 3D facial makeup estimation | makeup accuracy | Hurt |
| remove the alpha-matte regularization loss L′reg (ablation) | StyleGAN2 refinement · 3D facial makeup estimation | makeup accuracy | Hurt |
| use the PCA makeup prior instead of Yang-Res | 3D face reconstruction · BeautyFace | RMSE: 0.0545 -> 0.0690 | Hurt |
Surprising findings
- PCA model fine-tuning/refinement introduced artifacts rather than improving results, so refinement was applied only to the StyleGAN2 model.
- On BeautyFace, Yang-Res achieves a better HM(eyes) and RMSE than Ours (PCA), so the proposed PCA model is not uniformly best on every metric.
Limitations
- StyleGAN2-based estimation is computationally expensive (18.13s), with refinement dominating runtime.
- The PCA model retains only low-frequency makeup information and cannot reproduce fine details such as eyeshadow gradients.
- Makeup prior training data is small (3,070 images) and the StyleGAN2 approach struggles with uncommon makeup styles (out-of-domain).
Experimental setup
Datasets: 3,070 makeup UV images extracted by Yang-Ext from makeup datasets [30, 48] Wild BeautyFace
Benchmarks: Wild BeautyFace
Metrics: Histogram Matching (eyes/lips) RMSE SSIM LPIPS inference time
Baselines: Yang-Ext Yang-Res DECA Reconstruction module
Builds on: FLAME DECA StyleGAN2 pSp encoder Yang et al. 3D makeup estimation (Yang-Ext / Yang-Res)
Closest papers by meaning
- SHMT: Self-supervised Hierarchical Makeup Transfer via Latent Diffusion Models (NeurIPS 2024 · similarity 0.87)
- Content-Style Decoupling for Unsupervised Makeup Transfer without Generating Pseudo Ground Truth (CVPR 2024 · similarity 0.85)
- Diffusion Prior Interpolation for Flexibility Real-World Face Super-Resolution (AAAI 2025 · similarity 0.84)
- OSDFace: One-Step Diffusion Model for Face Restoration (CVPR 2025 · similarity 0.84)
- 3D Facial Expressions through Analysis-by-Neural-Synthesis (CVPR 2024 · similarity 0.84)
- StrokeFaceNeRF: Stroke-based Facial Appearance Editing in Neural Radiance Field (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.