Research map / Image generation / Editing and style transfer
LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
LumiNet transfers lighting from a target image to a source indoor scene by conditioning a latent diffusion ControlNet on latent intrinsics of the source and a latent lighting code of the target, trained on StyLitGAN-generated and real multi-illumination pairs.
Problem
Relighting indoor scenes by transferring lighting from one image to another, where spatial layouts and materials differ, without 3D reconstruction or multi-view inputs.
Approach
Operate ControlNet in latent intrinsic space (latent albedo-like features and low-dimensional lighting codes from a pretrained latent-intrinsic encoder), inject the target lighting code via an MLP adaptor into cross-attention of a latent diffusion model, and use variational StyLitGAN to generate paired relit training data.
Key results
- On MIIW, LumiNet achieves RMSE 0.180 and SSIM 0.647 on raw output versus 0.256 for RGB↔X (best baseline), a >20% improvement.
- Variational StyLitGAN reduces FID from 47.986 to 37.068, and CLIP filtering further lowers it to 35.810.
- In a 31-participant user study, LumiNet ranks first on image quality, lighting quality, and prompt alignment; its median surface normal angular error is 2.74 degrees versus 3.06 or more for baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LumiNet instead of RGB$X | Indoor scene relighting · MIIW test | RMSE (raw output): 0.256 -> 0.180 | Helped |
| Use LumiNet instead of RGB$X | Indoor scene relighting · MIIW test | SSIM (raw output): 0.476 -> 0.647 | Helped |
| Use variational StyLitGAN instead of standard StyLitGAN (ablation) | StyLitGAN data generation · StyLitGAN generated images | FID: 47.986 -> 37.068 | Helped |
| Add CLIP similarity filtering (ablation) | Variational StyLitGAN data generation · StyLitGAN generated images | FID: 37.068 -> 35.810 | Helped |
| Remove variational StyLitGAN-generated training data (ablation) | LumiNet · Indoor scene relighting | Recognition of scene-specific lighting effects | Hurt |
| Remove the latent intrinsic condition (ablation) | LumiNet · Indoor scene relighting | Relighting quality | Hurt |
| Remove the lighting adaptor and cross-attention fine-tuning (ablation) | LumiNet · Indoor scene relighting | Second-order lighting effects | Hurt |
| Remove flow-based inversion (ablation) | LumiNet · Indoor scene relighting | Artifact reduction | Hurt |
| Use only the latent intrinsic condition, without lighting cross-attention (ablation) | LumiNet · Indoor scene relighting | Second-order lighting effects | Hurt |
Surprising findings
- Plain ControlNet without latent intrinsic conditioning fails at relighting, producing an averaged color instead.
Limitations
- Struggles to detect small or occluded light sources and fails to capture chromatic shifts in dramatic lighting.
- Does not control light intensity, leading to brightness and color discrepancies.
- Relies on RF-inversion post-processing to reduce artifacts; trained only on same-scene pairs.
Experimental setup
Datasets: Multi-Illumination Images in the Wild (MIIW) BigTime LSUN Bedroom StyLitGAN-generated relit scenes
Benchmarks: MIIW (Multi-Illumination test set)
Metrics: RMSE SSIM FID CLIP-Score median angular error (surface normals) user study perceptual quality ranking
Baselines: Latent-Intrinsic RGB↔X IC-Light-v2 SA-AE S3Net ControlNet
Builds on: ControlNet Latent Intrinsics (Zhang et al. 2024) StyLitGAN Stable Diffusion RF-Inversion
Closest papers by meaning
- ScribbleLight: Single Image Indoor Relighting with Scribbles (CVPR 2025 · similarity 0.89)
- StyLitGAN: Image-Based Relighting via Latent Control (CVPR 2024 · similarity 0.88)
- Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset (CVPR 2025 · similarity 0.88)
- LightIt: Illumination Modeling and Control for Diffusion Models (CVPR 2024 · similarity 0.88)
- SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces (CVPR 2025 · similarity 0.88)
- UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.