Research map / Image generation / Image restoration
S^3-Face: SSS-Compliant Facial Reflectance Estimation via Diffusion Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Stable Diffusion v2 in two stages to estimate SSS-compliant facial reflectance (diffuse/specular albedo, normals, hemoglobin and melanin maps) from in-the-wild face images.
Problem
Monocular facial reflectance estimation is limited by small light-stage datasets with low subject diversity, and existing BRDF models do not capture subsurface scattering of skin.
Approach
Treat a pretrained latent diffusion model as an image-conditioned reflectance estimator, with a joint reflectance attention module for cross-domain geometric consistency, and a second diffusion net that predicts hemoglobin and melanin maps from the predicted diffuse albedo.
Key results
- Diffuse albedo PSNR 26.07 / SSIM 0.859 vs ID2Reflectance 24.13 / 0.782 on the captured test set.
- Two-stage training raises hemoglobin PSNR from 18.51 to 24.63 and melanin PSNR from 14.56 to 23.28 versus one-stage.
- Inference takes about 25 seconds per face on an A6000 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use S^3-Face instead of ID2Reflectance for diffuse albedo estimation | S^3-Face · Captured test set | PSNR: 24.13 -> 26.07 | Helped |
| Use S^3-Face instead of ID2Reflectance for diffuse albedo estimation | S^3-Face · Captured test set | SSIM: 0.782 -> 0.859 | Helped |
| Use S^3-Face instead of ID2Reflectance for specular albedo estimation | S^3-Face · Captured test set | PSNR: 22.67 -> 25.213 | Helped |
| Use S^3-Face instead of ID2Reflectance for specular albedo estimation | S^3-Face · Captured test set | SSIM: 0.771 -> 0.817 | Helped |
| Use S^3-Face instead of ID2Reflectance for normal estimation | S^3-Face · Captured test set | PSNR: 21.83 -> 24.48 | Helped |
| Use S^3-Face instead of ID2Reflectance for normal estimation | S^3-Face · Captured test set | SSIM: 0.723 -> 0.784 | Helped |
| Add two-stage training, predicting hemoglobin and melanin from diffuse albedo (ablation) | S^3-Face · Captured test set, hemoglobin | PSNR: 18.51 -> 24.63 | Helped |
| Add two-stage training, predicting hemoglobin and melanin from diffuse albedo (ablation) | S^3-Face · Captured test set, hemoglobin | SSIM: 0.6543 -> 0.7156 | Helped |
| Add two-stage training, predicting hemoglobin and melanin from diffuse albedo (ablation) | S^3-Face · Captured test set, melanin | PSNR: 14.56 -> 23.28 | Helped |
| Add two-stage training, predicting hemoglobin and melanin from diffuse albedo (ablation) | S^3-Face · Captured test set, melanin | SSIM: 0.4846 -> 0.6123 | Helped |
| Add joint reflectance attention between reflectance domains and color images (ablation) | S^3-Face DSN-Net · Reflectance estimation | reconstruction quality | Helped |
| Replace joint reflectance attention with cross-domain attention (ablation) | S^3-Face DSN-Net · Reflectance estimation | diffuse reconstruction quality | Hurt |
Surprising findings
- Fixed inference noise leads to mode collapse, while random noise gives more realistic results.
Limitations
- Training data has limited skin-tone diversity, supplemented with 142 albedo maps.
- Hemoglobin and melanin are sensitive to lighting, requiring a two-stage pipeline.
- Dataset is not released, limiting reproducibility.
Experimental setup
Datasets: Light Stage facial reflectance dataset (100 participants) 3D Scan Store albedo maps (142) VISIA skin analyzer images (128 patients, evaluation) LAION-5B (pretraining of SD, indirect)
Benchmarks: Light Stage captured test set Digital Emily VISIA skin analyzer patient data
Metrics: PSNR SSIM
Baselines: TRUST ID2Albedo ReflectanceMM ID2Reflectance AvatarMe++ Relightify Dib et al.
Builds on: Stable Diffusion latent diffusion models ID2Reflectance Deep3D v-prediction
Closest papers by meaning
- Monocular Identity-Conditioned Facial Reflectance Reconstruction (CVPR 2024 · similarity 0.90)
- A Morphable Face Albedo Model (CVPR 2020 · similarity 0.89)
- SFDM: Robust Decomposition of Geometry and Reflectance for Realistic Face Rendering from Sparse-view Images (CVPR 2025 · similarity 0.87)
- Diffusion Reflectance Map: Single-Image Stochastic Inverse Rendering of Illumination and Reflectance (CVPR 2024 · similarity 0.87)
- Diffusion Prior Interpolation for Flexibility Real-World Face Super-Resolution (AAAI 2025 · similarity 0.85)
- UV-IDM: Identity-Conditioned Latent Diffusion Model for Face UV-Texture Generation (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.