Research map / Point clouds / Shape and surface completion
FaceCom: Towards High-fidelity 3D Facial Shape Completion via Optimization and Inpainting Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FaceCom, a mesh-based GNN autoencoder shape generator fitted by latent optimization guided by a fine-tuned RePaint image inpainting prior, plus post-processing, for high-fidelity 3D facial shape completion of arbitrary incomplete scans.
Problem
High-fidelity 3D facial shape completion for incomplete, irregular facial scans (e.g., clinical facial defects), where point cloud and voxel methods trained on ShapeNet-style data give rough results.
Approach
Train a hypersphere-regularized mesh autoencoder on a mixed unified-topology facial dataset, then complete incomplete inputs by optimizing the latent code under a fitting loss plus an image-inpainting loss from a fine-tuned RePaint model rendered from the mesh, followed by normal-direction projection post-processing.
Key results
- Fitting on neutral faces: CD 0.160 mm with post-processing vs 0.578 mm for best baseline NPHM (w/o post-processing 0.480 mm), and MD 0.275 mm vs 0.309 mm for FLAME (lowest MD among baselines).
- Clinical nasal defect completion on 20 patients gives average margin fitness 0.33±0.21 mm.
- Ablation: full shape generator reconstruction 0.153 vs 0.336 for global-only and 0.208 for local-only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FaceCom's shape generator instead of NPHM | Neutral-face fitting · Neutral-face test set | CD (mm): 0.578 -> 0.480 | Helped |
| use FaceCom's shape generator instead of NPHM | Neutral-face fitting · Neutral-face test set | MD (mm): 0.381 -> 0.275 | Helped |
| add the local module (ablation) | FaceCom shape generator · Shape-generator training set | reconstruction loss: 0.336 -> 0.153 | Helped |
| add the local module (ablation) | FaceCom shape generator · Shape-generator training set | regularization loss: 0.240 -> 0.419 | Hurt |
| add the global module (ablation) | FaceCom shape generator · Shape-generator training set | reconstruction loss: 0.208 -> 0.153 | Helped |
| add the global module (ablation) | FaceCom shape generator · Shape-generator training set | regularization loss: 0.419 -> 0.419 | No clear effect |
| add image-inpainting guidance (ablation) | FaceCom · Large facial defect completion | visual completion quality | Helped |
Surprising findings
- Image inpainting guidance has minimal impact for small defects and only helps for large defect areas.
- Standard VAE training suffered vanishing KL and non-decreasing reconstruction loss, so a hypersphere norm regularizer was used instead.
Limitations
- Focuses on neutral faces with basic shape variation; expressions and texture are not modeled.
- No quantitative comparison for shape completion; only visual results and clinical margin fitness are reported.
- Larger defects yield greater discrepancy from ground truth due to limited information.
Experimental setup
Datasets: Headspace FaceScape in-house clinical neutral facial scans CelebA-HQ
Benchmarks: in-house facial scan test set (fitting) clinical nasal defect cases (20 patients)
Metrics: L2 chamfer distance mean point-to-surface distance margin fitness (RMS distance)
Baselines: FLAME FaceScape ImFace NPHM
Builds on: RePaint FeaStNet Ranjan et al. mesh autoencoder 3D morphable models PyTorch3D
Closest papers by meaning
- Geometry-Aware Face Completion and Editing (AAAI 2019 · similarity 0.87)
- Generative Face Completion (CVPR 2017 · similarity 0.86)
- DiffComplete: Diffusion-based Generative 3D Shape Completion (NeurIPS 2023 · similarity 0.86)
- ShapeFormer: Transformer-Based Shape Completion via Sparse Representation (CVPR 2022 · similarity 0.85)
- Deformable Shape Completion With Graph Convolutional Autoencoders (CVPR 2018 · similarity 0.85)
- LeGO: Leveraging a Surface Deformation Network for Animatable Stylized Face Generation with One Example (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.