Research map / 3D vision / Depth and stereo
Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Neural LightRig fine-tunes a Stable Diffusion model to generate nine consistent multi-light images from one input image, then uses a U-Net G-buffer model to predict normals and PBR materials from them.
Problem
Estimating surface normals and PBR materials (albedo, roughness, metallic) from a single image is severely under-constrained, and optimization-based or feed-forward methods lack accuracy and generalization, while photometric stereo requires costly capture setups.
Approach
Use a fine-tuned image diffusion model to synthesize multi-directional point-light images from a single input (arranged as a 3x3 grid with hybrid concatenation plus reference attention conditioning), then feed these pseudo photometric-stereo images with light poses into a U-Net regression model that predicts G-buffers, with augmentations to bridge the generated-to-rendered domain gap.
Key results
- Normal estimation mean angular error 6.413 vs 8.034 for the next best (StableNormal) on the LightProp held-out set.
- Albedo PSNR 26.62 and relighting PSNR 30.12 with SSIM 0.9601 and LPIPS 0.0371, outperforming IntrinsicAnything and IC-Light on relighting.
- Ablation: increasing multi-light input images from 0 to 9 reduces normal MAE from 7.563 to 4.059 degrees.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multi-light diffusion and G-buffer prediction | Neural LightRig · LightProp | mean angular error: 8.034 -> 6.413 | Helped |
| add multi-light diffusion and G-buffer prediction | Neural LightRig · LightProp | albedo PSNR: 23.88 -> 26.62 | Helped |
| add multi-light diffusion and G-buffer prediction | Neural LightRig · LightProp | relighting PSNR: 27.98 -> 30.12 | Helped |
| add reference attention to concatenation (ablation) | multi-light diffusion · Multi-light diffusion | PSNR: 19.32 -> 20.01 | Helped |
| add concatenation to reference attention (ablation) | multi-light diffusion · Multi-light diffusion | PSNR: 19.87 -> 20.01 | Helped |
| increase multi-light images from 0 to 3 (ablation) | large G-buffer model · LightProp | normal MAE: 7.563 -> 4.763 | Helped |
| increase multi-light images from 3 to 6 (ablation) | large G-buffer model · LightProp | normal MAE: 4.763 -> 4.275 | Helped |
| increase multi-light images from 6 to 9 (ablation) | large G-buffer model · LightProp | normal MAE: 4.275 -> 4.059 | Helped |
| add the proposed augmentation strategy (ablation) | large G-buffer model · LightProp | normal MAE: 7.080 -> 6.342 | Helped |
Limitations
- Evaluation mainly on synthetic held-out Objaverse objects; real-world quantitative accuracy is not measured.
- Fixed set of nine light positions; multi-light generation relies on the diffusion model's consistency.
- Hardware and training compute are not reported.
Experimental setup
Datasets: LightProp
Benchmarks: LightProp held-out subset
Metrics: mean angular error median angular error angular accuracy at 3-30 degrees PSNR RMSE SSIM LPIPS average time per frame
Baselines: DSINE GeoWizard Marigold StableNormal RGB↔X Yi et al. IntrinsicAnything DiLightNet IC-Light
Builds on: Stable Diffusion Zero123-style multi-view diffusion photometric stereo v-prediction diffusion
Closest papers by meaning
- Neural Gaffer: Relighting Any Object via Diffusion (NeurIPS 2024 · similarity 0.91)
- DiffusionLight: Light Probes for Free by Painting a Chrome Ball (CVPR 2024 · similarity 0.89)
- ReCap: Better Gaussian Relighting with Cross-Environment Captures (CVPR 2025 · similarity 0.88)
- Eclipse: Disambiguating Illumination and Materials using Unintended Shadows (CVPR 2024 · similarity 0.88)
- Intrinsic Image Diffusion for Indoor Single-view Material Estimation (CVPR 2024 · similarity 0.88)
- Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.