Research map / Image generation / Editing and style transfer
LightIt: Illumination Modeling and Control for Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds explicit lighting control to Stable Diffusion by conditioning a ControlNet-style module on normal and single-bounce direct shading maps, using a shading estimator to build a paired real image-shading dataset, and also applies the dataset to identity-preserving relighting.
Problem
Text-to-image diffusion models offer no explicit, consistent control over scene illumination, and lighting-annotated real-world data for training such control is scarce.
Approach
Estimate single-bounce direct shading (including cast shadows) from a single image via a density-field ray-marching model, use it to auto-generate a paired image/normal/shading dataset from outdoor HDR panoramas, and train a residual control encoder/decoder with a reconstruction loss to inject lighting into frozen Stable Diffusion.
Key results
- User study (42 participants) preferred the lighting-controlled model over SD for lighting consistency at 95.57% vs 4.43%.
- On relighting, the method reaches FID 64.18 and PSNR 20.44 versus OutCast's FID 71.08 and PSNR 19.79.
- Direct shading conditioning improves L-PSNR to 13.04 over N-dot-L shading at 6.43 on the control consistency test.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LightIt lighting conditioning | Stable Diffusion image generation · Image synthesis user study | L-PQ: 4.43 -> 95.57 | Helped |
| add LightIt lighting conditioning | Stable Diffusion image generation · Image synthesis user study | I-PQ: 39.14 -> 60.86 | Helped |
| add LightIt lighting conditioning | Stable Diffusion image generation · Image synthesis user study | T-PQ: 44.14 -> 55.86 | Helped |
| use LightIt diffusion-based relighting instead of OutCast | Image relighting · Image relighting test set | FID: 71.08 -> 64.18 | Helped |
| use LightIt diffusion-based relighting instead of OutCast | Image relighting · Image relighting test set | PSNR: 19.79 ± 4.39 -> 20.44 ± 3.34 | Helped |
| use LightIt diffusion-based relighting instead of OutCast | Image relighting · Image relighting user study | L-PQ: 54.27 -> 45.74 | Hurt |
| use LightIt diffusion-based relighting instead of OutCast | Image relighting · Image relighting user study | I-PQ: 36.47 -> 63.53 | Helped |
| add direct shading, including cast-shadow information, instead of N-dot-L shading (ablation) | LightIt image synthesis · Lighting control test set | L-PSNR: 6.43 ± 2.20 -> 13.04 ± 3.57 | Helped |
| add normal-map conditioning to direct shading (ablation) | LightIt image synthesis · Lighting control test set | N-PSNR: 17.30 ± 2.75 -> 17.47 ± 2.72 | Helped |
| add the Residual Control Decoder and control reconstruction loss (ablation) | LightIt relighting · Image relighting test set | FID: 76.28 -> 64.18 | Helped |
| replace the residual control architecture with ControlNet (ablation) | LightIt relighting · Identity preservation examples | identity preservation | Hurt |
| add a control decoder with reconstruction loss (ablation) | LightIt relighting · Identity preservation examples | identity preservation | Helped |
Surprising findings
- Conditioning on N-dot-L shading without cast shadows still conveys overall lighting but fails to produce realistic shadows.
- Standard ControlNet architecture tended to ignore part of the control signal, altering wall tint in relighting.
Limitations
- Assumes directional lighting, suited to outdoor scenes; point and other light sources are not handled.
- Shading estimation requires a known lighting direction for best results.
- Shadow regions carry no geometric signal, so normals are needed to avoid random geometry there.
Experimental setup
Datasets: Outdoor Laval Dataset OutCast relit images
Benchmarks: internal test set in-the-wild images
Metrics: L-PSNR L-AE N-PSNR N-AE PSNR FID user study preference (L-PQ, I-PQ, T-PQ)
Baselines: Stable Diffusion OutCast ControlNet
Builds on: ControlNet Stable Diffusion OutCast Outdoor Laval Dataset BLIP-2
Closest papers by meaning
- StyLitGAN: Image-Based Relighting via Latent Control (CVPR 2024 · similarity 0.88)
- IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation (NeurIPS 2025 · similarity 0.88)
- LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting (CVPR 2025 · similarity 0.88)
- UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting (NeurIPS 2025 · similarity 0.87)
- Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset (CVPR 2025 · similarity 0.87)
- SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.