Research map / 3D vision / Face and shape
Relightable Gaussian Codec Avatars
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds real-time relightable, animatable head avatars by combining 3D Gaussian splatting geometry decoded from a CVAE latent with a learned radiance-transfer appearance model (SH diffuse plus spherical Gaussian specular) and an explicit eye model.
Problem
Real-time relighting of animatable human head avatars with high fidelity for hair, skin, and eyes under all-frequency, continuous environment illumination; existing mesh/volumetric geometry and neural relighting models lack hair-strand detail and all-frequency reflections.
Approach
Parameterize 3D Gaussians on a shared UV map decoded by 2D CNNs from a CVAE expression latent, and learn per-Gaussian radiance transfer (SH diffuse with global light transport plus a view-dependent spherical Gaussian specular lobe with learned normals and visibility), so lighting is applied at the output rather than as an input to the network, preserving linearity and enabling all-frequency religh
Key results
- On held-out segments, the full model reaches PSNR 36.501, SSIM 0.943, LPIPS 0.110, outperforming MVP-based geometry (27.594 PSNR with EyeNeRF appearance) and the Linear appearance model (33.831 PSNR with the same geometry).
- On held-out lights, the full model reaches PSNR 34.042 and LPIPS 0.148, versus 32.829 PSNR for Linear [87] with the same geometry.
- Replacing the specular component with view-dependent SH (EyeNeRF) lowers held-out-segment PSNR from 36.529 to 35.110 when combined with the same geometry (rows C vs E).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed radiance-transfer appearance instead of EyeNeRF (ablation) | Ours geometry · Held-out segments | PSNR: 35.110 -> 36.501 | Helped |
| Use the proposed radiance-transfer appearance instead of Linear (ablation) | Ours geometry · Held-out segments | PSNR: 33.831 -> 36.529 | Helped |
| Add the explicit eye model (ablation) | Ours geometry with EyeNeRF appearance · Held-out segments | PSNR: 36.501 -> 34.550 | Hurt |
| Replace MVP geometry with the proposed Gaussian geometry (ablation) | EyeNeRF appearance · Held-out segments | PSNR: 27.594 -> 35.110 | Helped |
| Replace MVP geometry with the proposed Gaussian geometry (ablation) | Linear appearance · Held-out segments | PSNR: 36.294 -> 33.831 | Hurt |
| Replace MVP geometry with the proposed Gaussian geometry (ablation) | Proposed appearance · Held-out segments | PSNR: 35.789 -> 36.529 | Helped |
| Use the proposed radiance-transfer appearance instead of EyeNeRF (ablation) | Ours geometry · Held-out lights | PSNR: 30.836 -> 33.845 | Helped |
| Use the proposed radiance-transfer appearance instead of Linear (ablation) | Ours geometry · Held-out lights | PSNR: 32.829 -> 33.845 | Helped |
| Add the explicit eye model (ablation) | Ours geometry with EyeNeRF appearance · Held-out lights | PSNR: 30.836 -> 30.7976 | Hurt |
| Replace MVP geometry with the proposed Gaussian geometry (ablation) | EyeNeRF appearance · Held-out lights | PSNR: 28.030 -> 30.836 | Helped |
| Replace MVP geometry with the proposed Gaussian geometry (ablation) | Linear appearance · Held-out lights | PSNR: 33.444 -> 32.829 | Hurt |
| Replace MVP geometry with the proposed Gaussian geometry (ablation) | Proposed appearance · Held-out lights | PSNR: 33.778 -> 34.042 | Helped |
Surprising findings
- Linear relighting model achieves higher PSNR (36.294) than the full model (35.789) on held-out segments when paired with MVP geometry, although its LPIPS is worse.
- The approach works with a generic template mesh instead of expression-dependent geometry and remains effective after ablating up to 90% of cameras or 95% of light patterns.
Limitations
- Requires coarse mesh and gaze tracking as preprocessing, which can be sensitive to tracking failures.
- Not applicable in the wild due to lack of known illumination; relighting cost scales linearly with the number of avatars and per Gaussian.
Experimental setup
Datasets: in-house multiview light-stage capture (110 cameras, 460 LEDs)
Benchmarks: held-out segments (conversational expressions) held-out lights
Metrics: PSNR SSIM LPIPS
Baselines: MVP EyeNeRF Linear (Yang et al.)
Builds on: 3D Gaussian Splatting Precomputed Radiance Transfer Mixture of Volumetric Primitives Codec Avatars Explicit eye model (Schwartz et al.)
Closest papers by meaning
- BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading (NeurIPS 2025 · similarity 0.89)
- MeGA: Hybrid Mesh-Gaussian Head Avatar for High-Fidelity Rendering and Head Editing (CVPR 2025 · similarity 0.89)
- LUCAS: Layered Universal Codec Avatars (CVPR 2025 · similarity 0.88)
- Generalizable and Animatable Gaussian Head Avatar (NeurIPS 2024 · similarity 0.88)
- 3D Gaussian Head Avatars with Expressive Dynamic Appearances by Compact Tensorial Representations (CVPR 2025 · similarity 0.88)
- HRAvatar: High-Quality and Relightable Gaussian Head Avatar (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.