Research map / 3D vision / Face and shape
MoSAR: Monocular Semi-Supervised Model for Avatar Reconstruction using Differentiable Shading
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoSAR, a semi-supervised monocular avatar reconstruction method that combines light stage and in-the-wild images via a differentiable shading formulation with ambient occlusion and translucency, producing 4K relightable skin reflectance maps.
Problem
Single-portrait 3D avatar reconstruction requires disentangling geometry, reflectance and lighting; light stage data is accurate but costly and generalizes poorly to in-the-wild images.
Approach
Semi-supervised training that uses labeled light stage data plus unlabeled in-the-wild images, with a novel differentiable shading equation (Microfacet-extended, SH lighting, with ambient occlusion and subsurface translucency terms) so in-the-wild images supervise reflectance decomposition through self-supervised reconstruction.
Key results
- Ranks second on REALY with average error 1.500 mm, behind only HiFace trained with 200k synthetic images (1.275 mm) and ahead of HRN (1.537 mm).
- Full semi-supervised model reaches SSIM 0.83 diffuse, 0.65 specular, 0.85 AO, 0.82 translucency versus 0.69/0.28/0.61/0.67 for supervised-only.
- On 25 subjects, diffuse SSIM 0.754 vs 0.720 for FitMe and 0.704 for Relightify.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a non-linear morphable model instead of a linear 3DMM (ablation) | MoSAR geometry reconstruction · REALY | All-region reconstruction error (mm): 1.890 -> 1.500 | Helped |
| add light normalization to supervised training (ablation) | MoSAR texture estimation · Novel-light renders, specular map | SSIM: 0.28 -> 0.55 | Helped |
| add light normalization to supervised training (ablation) | MoSAR texture estimation · Novel-light renders, diffuse map | SSIM: 0.69 -> 0.80 | Helped |
| add light normalization to supervised training (ablation) | MoSAR texture estimation · Novel-light renders, AO map | SSIM: 0.61 -> 0.82 | Helped |
| add light normalization to supervised training (ablation) | MoSAR texture estimation · Novel-light renders, translucency map | SSIM: 0.67 -> 0.79 | Helped |
| remove light-stage supervision and train self-supervised only (ablation) | MoSAR texture estimation · Novel-light renders, specular map | SSIM: 0.65 -> 0.19 | Hurt |
| remove light-stage supervision and train self-supervised only (ablation) | MoSAR texture estimation · Novel-light renders, AO map | SSIM: 0.85 -> 0.36 | Hurt |
| remove light-stage supervision and train self-supervised only (ablation) | MoSAR texture estimation · Novel-light renders, diffuse map | SSIM: 0.83 -> 0.80 | Hurt |
| remove light-stage supervision and train self-supervised only (ablation) | MoSAR texture estimation · Novel-light renders, translucency map | SSIM: 0.82 -> 0.65 | Hurt |
| combine light-stage supervision with in-the-wild self-supervision (ablation) | MoSAR texture estimation · Novel-light renders, specular map | SSIM: 0.55 -> 0.65 | Helped |
| combine light-stage supervision with in-the-wild self-supervision (ablation) | MoSAR texture estimation · Novel-light renders, diffuse map | SSIM: 0.80 -> 0.83 | Helped |
| combine light-stage supervision with in-the-wild self-supervision (ablation) | MoSAR texture estimation · Novel-light renders, AO map | SSIM: 0.82 -> 0.85 | Helped |
Surprising findings
- Self-supervised-only training bakes most information into the diffuse map, with specular SSIM of only 0.19, showing that disentanglement needs labeled light stage data.
- The method tends to yield lighter skin tones for darker skin, attributed to FFHQ demographic imbalance.
Limitations
- Does not model external occluders such as eyeglasses and hair, which get baked into texture maps.
- Biased toward lighter skin tones due to FFHQ demographic imbalance.
- Geometry uses GNN decoder without explicit regularization beyond Laplacian smoothing; artifacts such as glasses baked into geometry.
Experimental setup
Datasets: light stage dataset (890 subjects) FFHQ FFHQ-UV REALY benchmark (HeadSpace scans) random HDR maps (Poly Haven)
Benchmarks: REALY light stage validation renders (SSIM)
Metrics: mean per-vertex L2 error (mm) SSIM
Baselines: MICA DECA PSL AlbedoGAN Deep3D HRN HiFace FitMe Relightify FaceVerse
Builds on: Microfacet BRDF DECA GNN auto-encoder mesh model (Anderson et al.) ESRGAN FFHQ-UV
Closest papers by meaning
- FRESA: Feedforward Reconstruction of Personalized Skinned Avatars from Few Images (CVPR 2025 · similarity 0.87)
- Vid2Avatar-Pro: Authentic Avatar from Videos in the Wild via Universal Prior (CVPR 2025 · similarity 0.86)
- High-Fidelity 3D Head Avatars Reconstruction through Spatially-Varying Expression Conditioned Neural Radiance Field (AAAI 2024 · similarity 0.86)
- RGBAvatar: Reduced Gaussian Blendshapes for Online Modeling of Head Avatars (CVPR 2025 · similarity 0.85)
- Relightable and Animatable Neural Avatars from Videos (AAAI 2024 · similarity 0.85)
- HRAvatar: High-Quality and Relightable Gaussian Head Avatar (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.