Research map / 3D vision / NeRF and radiance fields
ROGR: Relightable 3D Objects using Generative Relighting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a multi-view generative relighting diffusion model into a lighting-conditioned NeRF with dual (general and specular) illumination conditioning for fast feed-forward 3D object relighting.
Problem
Relightable 3D object reconstruction from multi-view images under unknown illumination; existing inverse rendering is brittle, and single-image relighting diffusion models produce view-inconsistent results and need per-illumination optimization when lifted to 3D.
Approach
Generate a multi-illumination, view-consistent relit dataset with a multi-view relighting diffusion model (CAT3D-style, trained on synthetic Objaverse), then train a NeRF-Casting-based relightable NeRF whose appearance is conditioned on a transformer embedding of the full environment map plus multi-scale blurred specular features sampled at reflection directions, enabling generalization to unseen
Key results
- On TensoIR, achieves PSNR 30.74, SSIM 0.950, LPIPS 0.069, improving over IllumiNeRF (29.71/0.947/0.072) and others.
- On Stanford-ORB, achieves best PSNR-H (26.21) and SSIM (0.980) and second-best PSNR-L (32.91) and LPIPS (0.027).
- Renders at 0.384 s per frame, comparable to R3DG (0.415 s) and much faster than optimization-per-light relighting methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ROGR rather than IllumiNeRF | 3D relighting · TensoIR | PSNR: 29.71 -> 30.74 | Helped |
| use ROGR rather than IllumiNeRF | 3D relighting · TensoIR | SSIM: 0.947 -> 0.950 | Helped |
| use ROGR rather than IllumiNeRF | 3D relighting · TensoIR | LPIPS: 0.072 -> 0.069 | Helped |
| use ROGR rather than Neural-PBIR | 3D relighting · Stanford-ORB | PSNR-H: 26.01 -> 26.21 | Helped |
| use ROGR rather than Neural-PBIR | 3D relighting · Stanford-ORB | PSNR-L: 33.26 -> 32.91 | Hurt |
| use ROGR rather than Neural-PBIR | 3D relighting · Stanford-ORB | SSIM: 0.979 -> 0.980 | Helped |
| use ROGR rather than Neural-PBIR | 3D relighting · Stanford-ORB | LPIPS: 0.023 -> 0.027 | Hurt |
| add multi-scale blurring to specular conditioning (ablation) | ROGR · TensoIR hotdog | PSNR: 31.35 -> 31.88 | Helped |
| add specular conditioning (ablation) | ROGR · TensoIR hotdog | PSNR: 30.00 -> 31.88 | Helped |
| add general conditioning (ablation) | ROGR · TensoIR hotdog | PSNR: 21.59 -> 31.88 | Helped |
| use environment-map conditioning instead of per-image appearance embeddings (ablation) | ROGR · TensoIR hotdog | PSNR: 19.12 -> 31.88 | Helped |
| use 512×512 rather than 128×128 environment maps (ablation) | ROGR · TensoIR hotdog | PSNR: 29.26 -> 31.88 | Helped |
Surprising findings
- Removing general conditioning causes large drops (PSNR 21.59), and per-image appearance embeddings perform worst (PSNR 19.12), showing per-illumination embeddings do not generalize.
- Removing specular blurring only mildly reduces metrics (PSNR 31.35 vs 31.88), while it still represents highly specular regions.
Limitations
- Training data lacks subsurface scattering, refraction and volumetric materials.
- Assumes distant environment-map lighting; near-field illumination not modeled.
- Object-centric; not extended to large-scale scene relighting.
Experimental setup
Datasets: Objaverse Polyhaven environment maps TensoIR Stanford-ORB
Benchmarks: TensoIR Stanford-ORB
Metrics: PSNR SSIM LPIPS PSNR-H PSNR-L
Baselines: NeRFactor InvRender PhySG NeRD NVDiffRec NVDiffRecMC Neural-PBIR NeRO TensoIR R3DG IllumiNeRF Neural Gaffer
Builds on: NeRF-Casting CAT3D Neural Gaffer DINO NeRF-in-the-Wild
Closest papers by meaning
- IllumiNeRF: 3D Relighting Without Inverse Rendering (NeurIPS 2024 · similarity 0.90)
- A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields (AAAI 2024 · similarity 0.90)
- ReCap: Better Gaussian Relighting with Cross-Environment Captures (CVPR 2025 · similarity 0.89)
- Neural Gaffer: Relighting Any Object via Diffusion (NeurIPS 2024 · similarity 0.89)
- Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation (CVPR 2025 · similarity 0.89)
- MetaGS: A Meta-Learned Gaussian-Phong Model for Out-of-Distribution 3D Scene Relighting (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.