Research map / Probabilistic and scientific modeling / Variational and generative sampling
Latent Intrinsics Emerge from Training to Relight
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a purely data-driven autoencoder on paired same-scene images under different lighting so that latent intrinsics and extrinsics emerge, enabling relighting and zero-shot albedo recovery.
Problem
Image relighting of real scenes: inverse-graphics pipelines need explicit geometry/material choices and struggle with indoor interreflections, and supervised relighting needs lighting labels or depth.
Approach
Relighting is done entirely with latent variables: an encoder yields spatial latent intrinsics from the target image and pooled latent extrinsics from a reference image, combined via a constrained scaling bottleneck (small alpha) and decoded. Intrinsicness is enforced by a coding-rate regularized invariance loss across lightings, and setting alpha=0 yields albedo without any albedo training data.
Key results
- On multi-illumination relighting with color correction, the model reaches RMSE 0.222 / SSIM 0.571, beating all baselines including light-supervised SA-AE (0.232 / 0.559).
- Zero-shot albedo from latent intrinsics on IIW reaches WHDR 15.81 with flattening, versus 22.61 for Intrinsic Diffusion, without any albedo or CG training data.
- Without color correction, only light-supervised SA-AE (RMSE 0.288) slightly beats the model (RMSE 0.297).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed latent relighting method instead of unsupervised SA-AE | MIT multi-illumination test · MIT multi-illumination test | RMSE (raw): 0.443 -> 0.297 | Helped |
| Use the proposed latent relighting method instead of light-supervised SA-AE | MIT multi-illumination test · MIT multi-illumination test | RMSE (raw): 0.288 -> 0.297 | Hurt |
| Apply global color correction to the proposed model output | Ours · MIT multi-illumination test | RMSE: 0.297 -> 0.222 | Helped |
| Disable denoising noise during training (σ=0) (ablation) | Ours · MIT multi-illumination test | SSIM (raw): 0.473 -> 0.232 | Hurt |
| Remove L_reg (ablation) | Ours · MIT multi-illumination test | SSIM (raw): 0.462 -> 0.473 | Hurt |
| Remove constrained scaling (α=∞) (ablation) | Ours · MIT multi-illumination test | RMSE (raw): 0.297 -> 0.471 | Hurt |
| Set α=1e-2 instead of α=5e-3 (ablation) | Ours · MIT multi-illumination test | RMSE (raw): 0.297 -> 0.314 | Hurt |
| Tune δ for WHDR evaluation (ablation) | Ours albedo · IIW test | WHDR: 28.97 -> 19.09 | Helped |
| Apply flattening postprocessing (ablation) | Ours albedo with tuned δ · IIW test | WHDR: 19.09 -> 15.81 | Helped |
| Use the proposed albedo estimate with flattening and tuned δ instead of Intrinsic Diffusion with flattening and tuned δ | IIW test · IIW test | WHDR: 17.10 -> 15.81 | Helped |
| Set α=5e-3 instead of α=1e-2 for albedo estimation (ablation) | Ours albedo · IIW test | WHDR: 15.81 -> 16.02 | Hurt |
Surprising findings
- Albedo emerges from relighting training alone: setting alpha=0 gives albedo estimates that beat a supervised CG-trained diffusion model on IIW WHDR.
- Unsupervised SA-AE leaks intrinsic information into its extrinsic code, while the constrained scaling bottleneck avoids this.
Limitations
- Requires paired images of the same scene under different lighting, which is resource-intensive to collect.
- Does not handle saturated pixel values common in LDR images.
- Latent intrinsics lack explicit physical meaning, limiting applications needing depth or normals.
Experimental setup
Datasets: MIT multi-illumination dataset IIW StyLitGAN generated images
Benchmarks: MIT multi-illumination dataset (relighting) IIW (albedo WHDR)
Metrics: RMSE SSIM WHDR
Baselines: SA-AE (supervised light index) SA-AE (unsupervised) S3-Net Intrinsic Diffusion Inverse Rendering BBA
Builds on: SA-AE StyLitGAN Intrinsic Diffusion
Closest papers by meaning
- IllumiNeRF: 3D Relighting Without Inverse Rendering (NeurIPS 2024 · similarity 0.89)
- Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation (CVPR 2025 · similarity 0.89)
- Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects (CVPR 2025 · similarity 0.89)
- StyLitGAN: Image-Based Relighting via Latent Control (CVPR 2024 · similarity 0.88)
- RobIR: Robust Inverse Rendering for High-Illumination Scenes (NeurIPS 2024 · similarity 0.87)
- UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.