Research map / Image generation / Image restoration
DiffusionLight: Light Probes for Free by Painting a Chrome Ball
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Estimates HDR lighting from a single image by inpainting a chrome ball with a depth-conditioned SDXL, using iterative noise-median inpainting and a LoRA for exposure bracketing.
Problem
Single-image HDR lighting estimation is limited by small, low-diversity HDR panorama datasets, so regressors generalize poorly to in-the-wild images.
Approach
Use a pre-trained LDR text-to-image diffusion model to inpaint a chrome ball into the input image, select good initial noise via iterative median-ball sampling (SDEdit), and fine-tune a LoRA on interpolated prompt embeddings to produce multiple exposures that are merged into an HDR map.
Key results
- On Laval Indoor three-sphere protocol, angular error is reduced vs. StyleLight by 49.5% (diffuse), 27.8% (matte), and 12.4% (mirror).
- On the array-of-spheres protocol the method gets si-RMSE 0.090 and angular error 5.25, outperforming EverLight and StyleLight but behind Weber et al. and EMLight.
- Ablation on Table 1 shows iterative inpainting and LoRA each improve over plain SDXL inpainting on most metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add iterative inpainting and LoRA to SDXL† (ablation) | SDXL† · Laval Indoor, three spheres, diffuse | si-RMSE: 0.20 -> 0.14 | Helped |
| Add iterative inpainting and LoRA to SDXL† (ablation) | SDXL† · Laval Indoor, three spheres, diffuse | Angular Error: 5.87 -> 2.14 | Helped |
| Add iterative inpainting and LoRA to SDXL† (ablation) | SDXL† · Poly Haven, three spheres, diffuse | si-RMSE: 0.22 -> 0.14 | Helped |
| Add iterative inpainting and LoRA to SDXL† (ablation) | SDXL† · Poly Haven, three spheres, diffuse | Angular Error: 2.98 -> 2.14 | Helped |
| Add iterative inpainting to SDXL† (ablation) | SDXL† · Laval Indoor, three spheres, diffuse | Angular Error: 5.87 -> 3.58 | Helped |
| Add LoRA to SDXL† (ablation) | SDXL† · Laval Indoor, three spheres, diffuse | Angular Error: 5.87 -> 3.47 | Helped |
| Add iterative inpainting to SDXL† (ablation) | SDXL† · Poly Haven, three spheres, diffuse | si-RMSE: 0.22 -> 0.16 | Helped |
| Add LoRA to SDXL† (ablation) | SDXL† · Poly Haven, three spheres, diffuse | Angular Error: 2.98 -> 2.35 | Helped |
| Combine iterative inpainting and LoRA (ablation) | SDXL† · Poly Haven, three spheres, matte | Angular Error: 3.56 -> 3.60 | Hurt |
| Use DiffusionLight instead of StyleLight | LDR environment map estimation · Laval Indoor | si-RMSE: 0.271 -> 0.303 | Hurt |
| Use DiffusionLight instead of StyleLight | LDR environment map estimation · Laval Indoor | RMSE: 0.246 -> 0.187 | Helped |
| Use DiffusionLight instead of StyleLight | LDR environment map estimation · Poly Haven | Angular Error: 6.291 -> 4.567 | Helped |
Surprising findings
- Initial diffusion noise maps consistently produce semantically similar inpainted objects (e.g., disco balls) across different input images.
Limitations
- Assumes orthographic projection when converting chrome ball to environment map, which may not match the diffusion model's projection.
- Chrome balls sometimes fail to reflect surroundings in overhead or bird's-eye-view images.
- Slow due to iterative inpainting and diffusion sampling.
Experimental setup
Datasets: Text2Light synthetic HDR panoramas Laval Indoor HDR Poly Haven
Benchmarks: Laval Indoor HDR Poly Haven
Metrics: scale-invariant RMSE angular error normalized RMSE
Baselines: StyleLight EverLight Weber et al. EMLight Blended Diffusion Paint-by-Example IP-Adapter DALL·E2 Adobe Firefly
Builds on: Stable Diffusion XL LoRA ControlNet SDEdit Text2Light
Closest papers by meaning
- Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion (CVPR 2025 · similarity 0.89)
- LuxDiT: Lighting Estimation with Video Diffusion Transformer (NeurIPS 2025 · similarity 0.89)
- Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models (CVPR 2025 · similarity 0.88)
- Neural Gaffer: Relighting Any Object via Diffusion (NeurIPS 2024 · similarity 0.88)
- Do Image Editing Models Understand Lighting? (NeurIPS 2026 · similarity 0.87)
- Diffusion Reflectance Map: Single-Image Stochastic Inverse Rendering of Illumination and Reflectance (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.