Research map / 3D vision / Face and shape
3D Face Reconstruction with the Geometric Guidance of Facial Part Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Part Re-projection Distance Loss (PRDL), which turns facial part segmentation into 2D point sets and matches anchor-based statistical distances to improve 3DMM reconstruction, especially for extreme expressions, and releases a synthetic expression dataset and a Part IoU benchmark.
Problem
3DMM-based 3D face reconstruction struggles with extreme expressions because landmark and photometric supervision are sparse or inaccurate, and renderer-based segmentation (silhouette IoU) losses suffer from local optima and unstable gradients.
Approach
Convert the segmentation of each facial part and the re-projected 3D mesh into 2D point sets, then compare grid-anchor-based statistical distances (nearest, furthest, average) between the two point sets, yielding a loss with clear gradients that avoids differentiable silhouette rendering.
Key results
- Ours achieves 73.82% average Part IoU, versus 71.69% for HRN and 69.51% for Deep3D, the best among compared methods.
- On REALY, Ours reaches 1.436 mm frontal-view and 1.442 mm side-view average 3D error, the lowest among compared methods.
- PRDL alone (w/o synthetic data) gives 72.22% Part IoU and 1.468 mm frontal error on the ablation, outperforming all other baselines in the table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PRDL (ablation) | Ours · Part IoU benchmark | Part IoU average (%): 72.08 -> 73.82 | Helped |
| add PRDL (ablation) | Ours · REALY frontal-view | 3D error average (mm): 1.568 -> 1.436 | Helped |
| add PRDL (ablation) | Ours · REALY side-view | 3D error average (mm): 1.528 -> 1.442 | Helped |
| add synthetic data (ablation) | Ours · Part IoU benchmark | Part IoU average (%): 72.22 -> 73.82 | Helped |
| add synthetic data (ablation) | Ours · REALY frontal-view | 3D error average (mm): 1.468 -> 1.436 | Helped |
| add synthetic data (ablation) | Ours · REALY side-view | 3D error average (mm): 1.455 -> 1.442 | Helped |
| use synthetic emotional-expression data without PRDL (ablation) | Ours · Part IoU benchmark | Part IoU average (%): 72.08 -> 72.22 | Helped |
| use PRDL without synthetic data (ablation) | Ours · Part IoU benchmark | Part IoU average (%): 72.08 -> 72.22 | Helped |
Surprising findings
- Lower 3D region error does not always correspond to better 2D part overlap; e.g., DECA has higher 2D eye IoU than 3DDFA-v2 despite a higher 3D forehead error.
Limitations
- Relies on the quality of an external face segmentation model (DML-CSR) and handles occluded parts by zeroing their loss weight.
- Part annotations on the 3D mesh are hand-crafted and require offline pre-completion for each topology.
Experimental setup
Datasets: Dad-3dheads CelebA CelebAMask-HQ RAF-ML RAF-DB 300W MEAD REALY
Benchmarks: Part IoU (MEAD subset) REALY
Metrics: IoU (%) 3D region error (mm)
Baselines: PRNet MGCNet Deep3D 3DDFA-v2 HRN DECA SoftRas DIB-R ReDA ICP chamfer distance
Builds on: 3DMM Deep3D DML-CSR SoftRas DIB-R
Closest papers by meaning
- Pixel3DMM: Versatile Screen-Space Priors for Single-Image 3D Face Reconstruction (ICLR 2026 · similarity 0.86)
- Sparse Photometric 3D Face Reconstruction Guided by Morphable Models (CVPR 2018 · similarity 0.86)
- 3D Face Morphable Models "In-The-Wild" (CVPR 2017 · similarity 0.86)
- MVF-Net: Multi-View 3D Face Morphable Model Regression (CVPR 2019 · similarity 0.86)
- Disentangling Features in 3D Face Shapes for Joint Face Reconstruction and Recognition (CVPR 2018 · similarity 0.86)
- OFER: Occluded Face Expression Reconstruction (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.