Research map / 3D vision / Face and shape
Monocular Identity-Conditioned Facial Reflectance Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns an image-space facial reflectance prior with multi-domain VQGAN codebooks trained on RGB and limited light-stage data, then injects identity via an AdaIN swapper to generate multi-view reflectance maps stitched into UV assets from a single image.
Problem
Monocular high-quality facial reflectance reconstruction is limited because light-stage reflectance datasets cover few subjects, so models generalize poorly to diverse in-the-wild identities.
Approach
Learn the reflectance prior in image space rather than UV space by jointly training a VQGAN with shared RGB and reflectance codebooks, fuse multi-domain codebooks, and perform identity-conditioned swapping trained in the RGB domain that transfers to the reflectance domain, then stitch three-view outputs into UV maps.
Key results
- Diffuse albedo PSNR of 28.47 versus 23.72 for ID2Albedo and 21.63 for TRUST on the captured test set, with identity similarity 0.735 versus 0.532.
- Multi-domain codebooks raise average reconstruction PSNR to 31.62 (diffuse) and 30.96 (specular) versus 24.87 and 20.95 for a joint codebook.
- Inference takes 3.8 seconds per face on an A6000 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-domain codebooks instead of a joint codebook (ablation) | ID2Reflectance · Captured test set, diffuse albedo | PSNR: 24.87 -> 31.62 | Helped |
| use multi-domain codebooks instead of a joint codebook (ablation) | ID2Reflectance · Captured test set, specular albedo | PSNR: 20.95 -> 30.96 | Helped |
| use multi-domain codebooks instead of a joint codebook (ablation) | ID2Reflectance · Captured test set, roughness | PSNR: 21.31 -> 31.59 | Helped |
| use multi-domain codebooks instead of a joint codebook (ablation) | ID2Reflectance · Captured test set, normal | PSNR: 20.56 -> 30.32 | Helped |
| select the closest reflectance template instead of using a fixed template (ablation) | ID2Reflectance · Captured test set, diffuse albedo | PSNR: 25.26 -> 28.47 | Helped |
| use ID2Reflectance instead of ID2Albedo | facial albedo estimation · Captured test set | PSNR: 23.72 -> 28.47 | Helped |
| use ID2Reflectance instead of ID2Albedo | facial albedo estimation · Captured test set | SSIM: 0.884 -> 0.923 | Helped |
| use ID2Reflectance instead of ID2Albedo | facial albedo estimation · CelebAMask-HQ | LPIPS: 0.1549 -> 0.1248 | Helped |
| use ID2Reflectance instead of ID2Albedo | facial albedo estimation · CelebAMask-HQ | ID similarity: 0.532 -> 0.735 | Helped |
Surprising findings
- Swapping trained only on RGB faces transfers to reflectance domains without reflectance-specific swapping training.
- Baselines SimSwap, InfoSwap and StyleGAN-based swappers fail or are restricted to texture and diffuse domains on reflectance maps.
Limitations
- Reflectance outputs are not supervised with ground-truth PBR maps and may not be physically plausible.
- Minor lipstick artifacts arise from the VGGFace2HQ restoration with GFPGAN.
- Skin tone gaps appear when source and target faces come from different races under a fixed template.
Experimental setup
Datasets: FFHQ private light-stage reflectance dataset (135 participants) VGGFace2HQ CelebAMask-HQ
Benchmarks: Digital Emily CelebAMask-HQ in-the-wild faces captured test set (20 participants)
Metrics: PSNR SSIM LPIPS identity similarity
Baselines: AvatarMe++ Relightify FitMe ReflectanceMM Dib et al. AlbedoMM TRUST ID2Albedo SimSwap InfoSwap E4S 3dSwap
Builds on: VQGAN ID2Albedo ArcFace projected GAN
Closest papers by meaning
- S^3-Face: SSS-Compliant Facial Reflectance Estimation via Diffusion Priors (CVPR 2025 · similarity 0.90)
- Monocular Reconstruction of Neural Face Reflectance Fields (CVPR 2021 · similarity 0.90)
- SFDM: Robust Decomposition of Geometry and Reflectance for Realistic Face Rendering from Sparse-view Images (CVPR 2025 · similarity 0.88)
- Diffusion Reflectance Map: Single-Image Stochastic Inverse Rendering of Illumination and Reflectance (CVPR 2024 · similarity 0.85)
- 3D Facial Expressions through Analysis-by-Neural-Synthesis (CVPR 2024 · similarity 0.85)
- MoSAR: Monocular Semi-Supervised Model for Avatar Reconstruction using Differentiable Shading (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.