Research map / 3D vision / Gaussian splatting
GAF: Gaussian Avatar Reconstruction from Monocular Videos via Multi-view Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reconstructs animatable 3D Gaussian head avatars from monocular videos by distilling priors from a normal-map-conditioned multi-view head latent diffusion model with iterative denoised pseudo-ground-truths and latent upsampling.
Problem
Photorealistic animatable head avatar reconstruction from monocular commodity-device videos is under-constrained because unobserved side/occluded regions lack supervision, causing artifacts in novel views.
Approach
Train a multi-view head latent diffusion model conditioned on a single input image (via VAE features and CLIP) and FLAME-rendered normal maps, and use its iteratively denoised outputs as pseudo-ground-truths for novel-view supervision of Gaussian avatars, instead of one-step SDS; a latent upsampler refines details.
Key results
- On NeRSemble novel view synthesis, GAF reaches LPIPS 0.125, PSNR 20.88, SSIM 88.91, best LPIPS and SSIM (PSNR second to GAGAvatar 21.20).
- On NeRSemble novel expression synthesis, GAF achieves LPIPS 0.087 and PSNR 24.12 versus best baseline GA 0.138 / 21.63.
- On monocular video dataset, GAF gets L1 0.0229, LPIPS 0.090, PSNR 23.16, SSIM 89.76, surpassing all baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GAF instead of GA | head avatar reconstruction · NeRSemble novel views | LPIPS: 0.218 -> 0.125 | Helped |
| use GAF instead of GA | head avatar reconstruction · NeRSemble novel expressions | PSNR: 21.63 -> 24.12 | Helped |
| use GAF instead of GA | head avatar reconstruction · Monocular Video | LPIPS: 0.109 -> 0.090 | Helped |
| add multi-view diffusion priors (ablation) | GAF · NeRSemble novel views | LPIPS: 0.207 -> 0.118 | Helped |
| use multi-view head diffusion instead of pretrained Stable Diffusion (ablation) | GAF · NeRSemble novel views | LPIPS: 0.196 -> 0.118 | Helped |
| use multi-view head diffusion instead of personalized Stable Diffusion (ablation) | GAF · NeRSemble novel views | LPIPS: 0.196 -> 0.118 | Helped |
| add iteratively denoised pseudo-ground truths (ablation) | GAF · NeRSemble novel views | LPIPS: 0.178 -> 0.118 | Helped |
| use normal-map conditioning instead of pose conditioning (ablation) | GAF · NeRSemble novel views | LPIPS: 0.131 -> 0.118 | Helped |
| use normal-map conditioning instead of raymap conditioning (ablation) | GAF · NeRSemble novel views | LPIPS: 0.150 -> 0.118 | Helped |
| add latent upsampler ×2 (ablation) | GAF · NeRSemble novel views | LPIPS: 0.134 -> 0.118 | Helped |
| replace iteratively denoised pseudo-ground truths with one-step SDS loss (ablation) | GAF · NeRSemble novel views | appearance saturation | Hurt |
Surprising findings
- Using pretrained or personalized Stable Diffusion priors gave worse PSNR than having no diffusion prior in the novel-view ablation.
- Vanilla one-step SDS loss caused appearance over-saturation in the face region, motivating the iterative pseudo-ground-truth approach.
Limitations
- Optimizing Gaussians with iteratively updated diffusion pseudo-ground-truths is time-consuming (about 12 hours per avatar).
- No explicit separation of material and appearance, so relighting is not supported.
- Quality limited by parametric FLAME head model, which lacks detailed hair geometry and animation.
Experimental setup
Datasets: NeRSemble INSTA monocular videos smartphone monocular captures
Benchmarks: NeRSemble INSTA monocular videos smartphone monocular captures
Metrics: LPIPS PSNR SSIM L1
Baselines: PanoHead SphereHead INSTA FlashAvatar GA P4D-v2 GAGAvatar
Builds on: GaussianAvatars ImageDream ReconFusion Stable Diffusion VHAP
Closest papers by meaning
- Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models (NeurIPS 2024 · similarity 0.90)
- GASP: Gaussian Avatars with Synthetic Priors (CVPR 2025 · similarity 0.90)
- Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians (CVPR 2024 · similarity 0.89)
- IM-Portrait: Learning 3D-aware Video Diffusion for Photorealistic Talking Heads from Monocular VideosC (CVPR 2025 · similarity 0.89)
- FATE: Full-head Gaussian Avatar with Textural Editing from Monocular Video (CVPR 2025 · similarity 0.89)
- RGBAvatar: Reduced Gaussian Blendshapes for Online Modeling of Head Avatars (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.