Research map / Image generation / Autoregressive and latent generation
Neural Gaffer: Relighting Any Object via Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Neural Gaffer fine-tunes Zero-1-to-3 on a synthetic relighting dataset (RelitObjaverse) so that a diffusion model relights a single object image under any HDR environment map, and uses it as a prior for two-stage NeRF relighting.
Problem
Single-image relighting of general objects under novel environment lighting, without explicit scene decomposition into geometry, materials, and lighting, which prior methods either restrict to specific categories or do poorly on.
Approach
Repurpose a pretrained image-conditioned novel-view diffusion model (Zero-1-to-3) as an end-to-end 2D relighting model by conditioning on the target-camera-aligned environment map encoded as two channels (tone-mapped LDR and log-normalized HDR), trained on a large synthetic object relighting dataset; the same model then serves as a prior for 3D NeRF relighting.
Key results
- On the single-image relighting validation set, Neural Gaffer reaches 26.706 PSNR / 0.927 SSIM / 0.041 LPIPS versus DiLightNet's 24.823 / 0.918 / 0.056.
- For 3D relighting, Neural Gaffer achieves 29.11 PSNR / 0.930 SSIM / 0.039 LPIPS, above NVDIFFREC-MC (28.25) and TensoIR (25.80).
- Ablations show removing the rotation step drops PSNR from 26.052 to 24.455 and removing the LDR map drops it to 25.503.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Neural Gaffer instead of DiLightNet | single-image relighting · Objaverse validation | PSNR: 24.823 -> 26.706 | Helped |
| use Neural Gaffer instead of DiLightNet | single-image relighting · Objaverse validation | SSIM: 0.918 -> 0.927 | Helped |
| use Neural Gaffer instead of DiLightNet | single-image relighting · Objaverse validation | LPIPS: 0.056 -> 0.041 | Helped |
| use Neural Gaffer instead of DiLightNet | single-image relighting · Objaverse validation, channel-aligned | PSNR: 26.931 -> 29.829 | Helped |
| use Neural Gaffer instead of DiLightNet | single-image relighting · Objaverse validation, channel-aligned | SSIM: 0.926 -> 0.939 | Helped |
| use Neural Gaffer instead of DiLightNet | single-image relighting · Objaverse validation, channel-aligned | LPIPS: 0.050 -> 0.028 | Helped |
| use Neural Gaffer instead of the strongest baseline, NVDIFFREC-MC | 3D radiance-field relighting · 3D relighting test objects | PSNR: 28.25 -> 29.11 | Helped |
| use Neural Gaffer instead of the strongest baseline, NVDIFFREC-MC | 3D radiance-field relighting · 3D relighting test objects | SSIM: 0.915 -> 0.930 | Helped |
| use Neural Gaffer instead of the strongest baseline, NVDIFFREC-MC | 3D radiance-field relighting · 3D relighting test objects | LPIPS: 0.082 -> 0.039 | Helped |
| add the LDR environment map (ablation) | Neural Gaffer · Relighting ablation benchmark | PSNR: 25.503 -> 26.052 | Helped |
| add the normalized HDR environment map (ablation) | Neural Gaffer · Relighting ablation benchmark | PSNR: 25.822 -> 26.052 | Helped |
| rotate the environment map explicitly instead of conditioning on relative 3D rotation (ablation) | Neural Gaffer · Relighting ablation benchmark | PSNR: 24.455 -> 26.052 | Helped |
Surprising findings
- Rotating the environment map explicitly into the target camera frame works better than encoding relative 3D rotation as an embedding, as in Zero-1-to-3.
Limitations
- Trained at 256x256 resolution; the VAE struggles to preserve fine object details, causing relighting failures.
- Color ambiguity between material and lighting for diffuse objects leads to incorrect relit colors.
- Misses high-frequency reflections in some relit images.
Experimental setup
Datasets: Objaverse RelitObjaverse NeRF-Synthetic
Benchmarks: RelitObjaverse validation (48 objects) NeRF-Synthetic and Objaverse 3D relighting test (6 objects)
Metrics: PSNR SSIM LPIPS channel-aligned PSNR/SSIM/LPIPS
Baselines: DiLightNet IC-Light TensoIR NVDIFFREC-MC SDS
Builds on: Zero-1-to-3 Stable Diffusion Objaverse TensoRF Score Distillation Sampling
Closest papers by meaning
- Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion (CVPR 2025 · similarity 0.91)
- SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces (CVPR 2025 · similarity 0.90)
- UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting (NeurIPS 2025 · similarity 0.90)
- IllumiNeRF: 3D Relighting Without Inverse Rendering (NeurIPS 2024 · similarity 0.89)
- MetaGS: A Meta-Learned Gaussian-Phong Model for Out-of-Distribution 3D Scene Relighting (NeurIPS 2025 · similarity 0.89)
- ROGR: Relightable 3D Objects using Generative Relighting (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.