Research map / 3D vision / NeRF and radiance fields
IllumiNeRF: 3D Relighting Without Inverse Rendering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Relights each input view with a ControlNet-style image diffusion model conditioned on radiance cues from estimated geometry, then distills multiple sampled relit images into a latent-code NeRF for novel-view relighting without inverse rendering.
Problem
Relightable view synthesis from images under unknown lighting; existing inverse rendering methods are brittle, ambiguous, and computationally expensive due to differentiable Monte Carlo rendering.
Approach
Replace inverse rendering with sampling multiple plausible relit images per view from a single-image relighting diffusion model (conditioned on geometry-based radiance cues under the target light), then fit a latent-code NeRF to these samples so that per-sample latent codes absorb material/explanation variation.
Key results
- On TensoIR, achieves PSNR 29.709, SSIM 0.947, LPIPS 0.072, outperforming TensoIR (28.580 / 0.944 / 0.081).
- On Stanford-ORB, achieves PSNR-H 25.42, PSNR-L 32.62, SSIM 0.976, LPIPS 0.027, beating all baselines except Neural-PBIR (26.01 PSNR-H).
- Ablation on TensoIR hotdog: latent NeRF with S=16 samples reaches PSNR 27.950 vs 24.957 for a standard NeRF with S=1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IllumiNeRF instead of TensoIR | Relightable 3D reconstruction · TensoIR | PSNR: 28.580 -> 29.709 | Helped |
| Use IllumiNeRF instead of Neural-PBIR | Relightable 3D reconstruction · Stanford-ORB | PSNR-L: 33.26 -> 32.62 | Hurt |
| Use a latent NeRF instead of a standard NeRF (ablation) | IllumiNeRF · TensoIR Hotdog | PSNR: 24.957 -> 26.321 | Helped |
| Increase diffusion samples from S=1 to S=4 (ablation) | IllumiNeRF latent NeRF · TensoIR Hotdog | PSNR: 26.321 -> 27.409 | Helped |
| Increase diffusion samples from S=4 to S=16 (ablation) | IllumiNeRF latent NeRF · TensoIR Hotdog | PSNR: 27.409 -> 27.950 | Helped |
Surprising findings
- Neural-PBIR scores higher on Stanford-ORB metrics but its relit results are mostly diffuse, lacking the specular highlights the method recovers.
- Stanford-ORB per-image illumination probes are inconsistent with the true incident lighting, so metrics penalize methods that recover specular highlights.
Limitations
- Depends on high-quality geometry from UniSDF; missing structure leads to missed specular reflections and thin-structure errors.
- Not suitable for real-time relighting since diffusion sampling and NeRF optimization are needed per target light.
- Stanford-ORB per-image illumination mismatch makes metrics unreliable for specular-heavy methods; performs worse than competitors under per-image reference.
Experimental setup
Datasets: Objaverse Polyhaven environment maps TensoIR Stanford-ORB
Benchmarks: TensoIR Stanford-ORB
Metrics: PSNR SSIM LPIPS PSNR-H PSNR-L
Baselines: NeRFactor InvRender TensoIR PhySG NVDiffRec NeRD NVDiffRecMC Neural-PBIR
Builds on: Stable Diffusion ControlNet UniSDF NeRF-Casting DiLightNet
Closest papers by meaning
- A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields (AAAI 2024 · similarity 0.91)
- ROGR: Relightable 3D Objects using Generative Relighting (NeurIPS 2025 · similarity 0.90)
- Few-Shot Neural Radiance Fields under Unconstrained Illumination (AAAI 2024 · similarity 0.90)
- Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation (CVPR 2025 · similarity 0.90)
- UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting (NeurIPS 2025 · similarity 0.90)
- Neural Gaffer: Relighting Any Object via Diffusion (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.