Research map / 3D vision / Face and shape
Learning to Decouple the Lights for 3D Face Texture Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Light Decoupling, which models unnatural illumination from external occlusions as multiple separate spherical-harmonic light conditions mixed by learned spatial-temporal neural masks, to recover clean 3D face textures from images and videos.
Problem
Recovering accurate 3D facial textures from faces under complicated, unnatural illumination caused by external occlusions (e.g., hats, hair), where single uniform illumination assumptions bake shadows into textures.
Approach
Replace the single-illumination assumption with n separate spherical-harmonic light conditions whose spatial-temporal masks are predicted by MLPs and adaptively pruned (ACE), plus a face-region mask network and realistic priors (global/local texture priors and a FaceNet-based human prior) for texture optimization.
Key results
- On VoxCeleb2 single images, the method reaches target-image PSNR 29.22 and LPIPS 6.36, versus 27.23 PSNR for the best baseline D3DFR.
- On video sequences, target PSNR is 29.15 versus 26.20 for D3DFR (the next best).
- On diverse-shadow images, PSNR is 28.97 versus 25.02 for D3DFR on the second metric block; ablations show removing light and occlusion networks drops PSNR from 29.22 to 25.19.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Light Decoupling | 3D face texture modeling · VoxCeleb2 single images, target | PSNR: 27.23 -> 29.22 | Helped |
| use Light Decoupling | 3D face texture modeling · VoxCeleb2 video sequences, target | PSNR: 26.20 -> 29.15 | Helped |
| use Light Decoupling | 3D face texture modeling · images with diverse shadows, second metric group | PSNR: 25.02 -> 28.97 | Helped |
| add global prior loss GP (ablation) | Ours · VoxCeleb2 single images | PSNR: 27.20 -> 28.81 | Helped |
| add local prior loss LP to GP (ablation) | Ours · VoxCeleb2 single images | PSNR: 28.81 -> 28.82 | Helped |
| add human prior loss HP to GP+LP (ablation) | Ours · VoxCeleb2 single images | PSNR: 28.82 -> 29.22 | Helped |
| add light representation f (ablation) | Ours · VoxCeleb2 single images | PSNR: 25.19 -> 27.52 | Helped |
| add occlusion representation g (ablation) | Ours · VoxCeleb2 single images | PSNR: 25.19 -> 29.22 | Helped |
| replace multiple light conditions with one global SH (ablation) | Ours · VoxCeleb2 single images | PSNR: 29.22 -> 25.87 | Hurt |
| vary initial lighting-condition count from 5 to 3 (ablation) | Ours · VoxCeleb2 single images | PSNR: 29.22 -> 28.49 | Hurt |
| vary initial lighting-condition count from 5 to 7 or 9 (ablation) | Ours · VoxCeleb2 single images | PSNR: 29.22 -> 29.14; 29.22 -> 29.16 | Hurt |
| replace human prior HP with perceptual loss (ablation) | Ours · VoxCeleb2 single images | PSNR: 29.22 -> 28.72 | Hurt |
Surprising findings
- Increasing the number of bands in a single global spherical harmonic gives only limited gains, suggesting external shadows are better modeled by multiple local light conditions.
Limitations
- Initialization relies on AlbedoMM, a linear model that may give inaccurate initial textures.
- Optimization over multiple light conditions is slower than single-illumination methods.
- Fine high-frequency details such as wrinkles may be lost.
Experimental setup
Datasets: VoxCeleb2 CelebAMask-HQ
Benchmarks: VoxCeleb2 single images VoxCeleb2 video sequences CelebAMask-HQ images with diverse shadows
Metrics: PSNR SSIM LPIPS
Baselines: CPEM D3DFR NextFace FFHQ-UV Deocclusion method
Builds on: NextFace AlbedoMM 3D Morphable Model FFHQ-UV
Closest papers by meaning
- SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces (CVPR 2025 · similarity 0.86)
- GroomLight: Hybrid Inverse Rendering for Relightable Human Hair Appearance Modeling (CVPR 2025 · similarity 0.85)
- SFDM: Robust Decomposition of Geometry and Reflectance for Realistic Face Rendering from Sparse-view Images (CVPR 2025 · similarity 0.85)
- Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion (CVPR 2025 · similarity 0.85)
- Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation (CVPR 2025 · similarity 0.84)
- Relightable and Animatable Neural Avatars from Videos (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.