Research map / 3D vision / Face and shape
Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Relights all input images jointly to a reference illumination with a multiview diffusion model, then reconstructs the object with a NeRF-Casting-based radiance field using per-image shading embeddings.
Problem
3D reconstruction of objects, especially specular ones, from photos captured under extremely varying illumination; per-image appearance embeddings and inverse rendering tend to explain away view-dependent effects and produce diffuse results.
Approach
Jointly relight a set of differently lit input images to one reference image's illumination using a multiview latent diffusion model (CAT3D-style 3D self-attention with raymaps), then fit a NeRF-Casting radiance field robust to residual inconsistencies via a per-image shading embedding that perturbs surface normals.
Key results
- On Objaverse shiny assets, the method reaches 26.54 PSNR / 0.911 SSIM / 0.090 LPIPS versus 23.46 PSNR for IllumiNeRF given ground-truth lighting.
- On 18 NAVI real scenes, it achieves 25.55 PSNR vs 24.01 for NeROIC and 23.59 for NeRF-Casting + AE.
- Relighting with 64 jointly predicted frames improves over 1 frame (standard assets PSNR 31.34 vs 28.39); standard appearance embeddings hurt (24.38 vs 26.02 with no embeddings).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use joint multiview relighting and robust radiance-field reconstruction | Ours vs IllumiNeRF w/ known light · Objaverse standard assets | PSNR: 29.22 -> 31.34 | Helped |
| Use joint multiview relighting and robust radiance-field reconstruction | Ours vs IllumiNeRF w/ known light · Objaverse shiny assets | PSNR: 23.46 -> 26.54 | Helped |
| Use the proposed reconstruction approach | Ours vs NeROIC · NAVI | PSNR: 24.01 -> 25.55 | Helped |
| Use the proposed reconstruction approach | Ours vs NeROIC · NAVI | SSIM: 0.918 -> 0.929 | Helped |
| Use the proposed reconstruction approach | Ours vs NeRF-Casting + AE · NAVI | LPIPS: 0.066 -> 0.060 | Helped |
| Increase simultaneously relit frames from 1 to 64 (ablation) | Ours · Objaverse standard assets | PSNR: 28.39 -> 31.34 | Helped |
| Increase simultaneously relit frames from 1 to 64 (ablation) | Ours · Objaverse shiny assets | PSNR: 23.85 -> 26.54 | Helped |
| Add shiny assets to training data (ablation) | Ours · Objaverse standard assets | PSNR: 29.16 -> 29.26 | Helped |
| Add shiny assets to training data (ablation) | Ours · Objaverse shiny assets | PSNR: 22.34 -> 22.91 | Helped |
| Add per-image shading embeddings (ablation) | NeRF-Casting reconstruction of harmonized shiny scenes · Objaverse shiny assets | PSNR: 26.02 -> 26.54 | Helped |
| Replace shading embeddings with appearance embeddings (ablation) | NeRF-Casting reconstruction of harmonized shiny scenes · Objaverse shiny assets | PSNR: 26.54 -> 24.38 | Hurt |
Surprising findings
- Adding purely specular (mirror) training data also improves performance on standard mostly-diffuse assets.
- Standard per-image appearance embeddings degrade reconstruction below using no embeddings, as they explain specularities as per-image diffuse color.
Limitations
- Requires object masks and accurate camera poses as inputs.
- Camera pose estimation is hard for highly reflective objects without reliable features.
Experimental setup
Datasets: Objaverse NAVI Poly Haven environment maps HDRIMaps proprietary ~300K high-quality object asset set
Benchmarks: Objaverse (standard and shiny assets) NAVI
Metrics: PSNR SSIM LPIPS
Baselines: NeRF-Casting + AE NeROIC IllumiNeRF
Builds on: NeRF-Casting CAT3D Stable Diffusion 1.5 NeRF-W
Closest papers by meaning
- RelitLRM: Generative Relightable Radiance for Large Reconstruction Models (ICLR 2025 · similarity 0.91)
- IllumiNeRF: 3D Relighting Without Inverse Rendering (NeurIPS 2024 · similarity 0.90)
- ReCap: Better Gaussian Relighting with Cross-Environment Captures (CVPR 2025 · similarity 0.90)
- Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects (CVPR 2025 · similarity 0.89)
- Eclipse: Disambiguating Illumination and Materials using Unintended Shadows (CVPR 2024 · similarity 0.89)
- Latent Intrinsics Emerge from Training to Relight (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.