Research map / 3D vision / Face and shape
High-Quality Facial Geometry and Appearance Capture at Home
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Captures complete relightable face assets (skin, mouth interior, hair, eyes) from a single co-located smartphone flashlight video in a dim room, using a hybrid eyeball-mesh plus neural SDF/BRDF representation with a combined lighting model and an AlbedoMM reflectance prior.
Problem
High-quality facial geometry and appearance capture is studio-bound or requires special setups (sunlight, polarization, darkroom); existing home-friendly methods only capture facial skin and struggle with highly reflective eyeballs.
Approach
A hybrid representation that models eyeballs as two predefined sphere meshes with fixed specular lobes and the rest of the face with a neural SDF plus neural Disney BRDF field, trained with a mesh-aware volume rendering scheme, a composition loss, and an AlbedoMM-based specular reflectance prior, under a combined point-flashlight plus SH ambient lighting model.
Key results
- On face reconstruction averaged over 5 subjects, the method reaches 26.12 PSNR / 0.8808 SSIM / 0.1642 LPIPS versus 17.62 / 0.7339 / 0.2727 for NextFace++.
- The combined lighting model improves PSNR over a point-light-only baseline by 0.08 dB (night w/ curtain), 0.05 dB (noon w/ curtain), and 0.61 dB (noon w/o curtain).
- Training completes within 70 minutes on a single RTX 3090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed hybrid face representation instead of NextFace++ | Face reconstruction · Hold-out validation images, 5 subjects | PSNR: 17.62 -> 26.12 | Helped |
| Use the proposed hybrid face representation instead of NextFace++ | Face reconstruction · Hold-out validation images, 5 subjects | SSIM: 0.7339 -> 0.8808 | Helped |
| Use the proposed hybrid face representation instead of NextFace++ | Face reconstruction · Hold-out validation images, 5 subjects | LPIPS: 0.2727 -> 0.1642 | Helped |
| Add eyeball mesh priors to the neural-field face representation (ablation) | Face reconstruction · Face geometry around eyes | geometry reconstruction quality | Helped |
| Add composition loss Lcomp to the hybrid representation (ablation) | Hybrid face representation · Face geometry around eyes | geometry reconstruction quality | Helped |
| Add eyeball geometry and reflectance priors (ablation) | Face relighting · Eye relighting | plausibility of specularities | Helped |
| Add ambient illumination to the point-light lighting model (ablation) | Combined lighting model · Night with curtain | PSNR: 0.08dB | Helped |
| Add ambient illumination to the point-light lighting model (ablation) | Combined lighting model · Noon with curtain | PSNR: 0.05dB | Helped |
| Add ambient illumination to the point-light lighting model (ablation) | Combined lighting model · Noon without curtain | PSNR: 0.61dB | Helped |
| Add AlbedoMM-based reflectance regularization Lref (ablation) | Reflectance estimation · Diffuse and specular albedo estimation | albedo quality | Helped |
| Use the AlbedoMM reflectance prior instead of a naive smooth prior (ablation) | Reflectance estimation · Diffuse and specular albedo estimation | albedo quality | Helped |
Surprising findings
- Simply overriding eyeball specular albedo and roughness on exported UV maps (holistic rep. w/ naive ref. prior) still fails to produce plausible eye specularities, because the geometry is wrong.
Limitations
- Eyeball position and radius are set manually.
- Lower quality than Light Stage: diffuse albedo less cleanly disentangled and no pore-level detail due to smartphone camera.
- Subject motion over the ~25-second capture can blur texture or bake eyelids into eyeball texture.
Experimental setup
Datasets: self-captured smartphone flashlight sequences
Benchmarks: hold-out validation frames from captured sequences
Metrics: PSNR SSIM LPIPS
Baselines: NextFace++ WildLight NeRO Light Stage-based solution (Ghosh et al.) holistic representation
Builds on: VolSDF Instant-NGP multi-resolution hash grid WildLight EyeNeRF AlbedoMM
Closest papers by meaning
- [Re] Lifting 2D StyleGAN for 3D-Aware Face Generation (NeurIPS 2022 · similarity 0.81)
- Learning Formation of Physically-Based Face Attributes (CVPR 2020 · similarity 0.80)
- Monocular Real-Time Full Body Capture With Inter-Part Correlations (CVPR 2021 · similarity 0.80)
- Modeling Facial Geometry Using Compositional VAEs (CVPR 2018 · similarity 0.79)
- U-VAP: User-specified Visual Appearance Personalization via Decoupled Self Augmentation (CVPR 2024 · similarity 0.79)
- Generalizable Face Landmarking Guided by Conditional Face Warping (CVPR 2024 · similarity 0.78)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.