Research map / 3D vision / Face and shape
Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Holo-Relighting performs controllable volumetric portrait relighting from a single image by relighting EG3D tri-plane features (from GAN inversion) conditioned on environment map and head pose, trained on light-stage OLAT data.
Problem
Simultaneous novel-view synthesis and novel-lighting control for a single in-the-wild portrait image, including head-pose-dependent lighting effects such as cast shadows and specular highlights, without explicit physical reflectance/lighting models.
Approach
Invert a pretrained EG3D to obtain 3D-aware GAN features from a delit albedo image, then use a coarse-to-fine relighting network conditioned on environment map and head pose to predict a shading tri-plane added to the albedo tri-plane; training uses multi-view inversion regularization and a quotient-image-style shading transfer to create pseudo ground truth.
Key results
- Free-view relighting on OLAT test set: LPIPS 0.0917, PSNR 27.35, SSIM 0.8511, outperforming FaceLit (PSNR 20.86) and NeRFFaceLighting (PSNR 19.40).
- 2D relighting vs Total Relighting: LPIPS 0.0981 vs 0.1997 and PSNR 26.17 vs 19.58.
- Ablation: multi-view inversion and shading transfer together reduce LPIPS from 0.1443 (single-view) to 0.0997 and raise PSNR from 23.57 to 25.89.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Holo-Relighting instead of FaceLit | Free-view portrait relighting · Free-view relighting test set | LPIPS: 0.1542 -> 0.0917 | Helped |
| use Holo-Relighting instead of Total Relighting (TR) | 2D portrait relighting · 2D relighting OLAT test set | LPIPS: 0.1997 -> 0.0981 | Helped |
| add multi-view inversion rather than relying on single-view inversion (ablation) | Holo-Relighting training data creation · Ablation test set | LPIPS: 0.1443 -> 0.0997 | Helped |
| add shading transfer (ablation) | Holo-Relighting training data creation · Ablation test set | LPIPS: 0.1382 -> 0.0997 | Helped |
Limitations
- Relies on GAN inversion quality; imperfect inversion causes high-frequency misalignment (addressed only partially by shading transfer)
- Limitations discussed only in supplemental file, not in main text
Experimental setup
Datasets: light stage OLAT captures (69 subjects, 931 sequences) PolyHaven environment maps (671)
Benchmarks: OLAT test set (235 images for free-view; 940 images for 2D relighting) in-the-wild portraits
Metrics: LPIPS NIQE Deg (LightCNN identity similarity) PSNR SSIM
Baselines: NeRFFaceLighting FaceLit Total Relighting
Builds on: EG3D Total Relighting NeRFFaceLighting FaceLit ControlNet
Closest papers by meaning
- SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces (CVPR 2025 · similarity 0.86)
- ROGR: Relightable 3D Objects using Generative Relighting (NeurIPS 2025 · similarity 0.86)
- SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting (CVPR 2024 · similarity 0.86)
- IllumiNeRF: 3D Relighting Without Inverse Rendering (NeurIPS 2024 · similarity 0.85)
- Real-time 3D-aware Portrait Video Relighting (CVPR 2024 · similarity 0.85)
- MetaGS: A Meta-Learned Gaussian-Phong Model for Out-of-Distribution 3D Scene Relighting (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.