Research map / Image generation / Image restoration

LuxDiT: Lighting Estimation with Video Diffusion Transformer

NeurIPS 2025 · Image generation › Image restoration · approach: Image harmonization and color transformation · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Fine-tunes the CogVideoX video diffusion transformer to generate HDR environment maps from single images or videos, using synthetic renderings, HDR panoramas, and LoRA adaptation on real panoramas.

Problem

Estimating HDR scene illumination (environment maps) from a single LDR image or video is hard because real HDR lighting ground truth is scarce, and lighting requires global, non-local reasoning and high dynamic range output.

Approach

Treat lighting estimation as conditional generation with a video DiT: input image/video tokens are attended jointly with denoising tokens of dual-tonemapped (Reinhard and log) HDR panorama latents, trained on synthetic renders and then LoRA-adapted on real HDR panoramas to improve semantic alignment.

Key results

• On Laval Outdoor sunlight scenes, peak angular error mean 23.7 vs 44.4 for DiffusionLight (about 47% reduction).
• On Poly Haven, angular error diffuse 1.235 vs 2.199 for DiffusionLight (best across three sphere materials).
• Virtual object insertion: Ours preferred over StyleLight and DiffusionLight in 60.6% of user comparisons; comparable to DiPIR (54.5%).

Measured effects reported in the paper

Surprising findings

• LoRA scale helps on real photos but hurts lighting accuracy on synthetic foreground objects, where larger LoRA scale causes foreground content to appear in the predicted environment map.
• Without synthetic pretraining the model does well in-domain (Poly Haven) but degrades out-of-domain on Laval Indoor.

Limitations

• Inference is computationally intensive due to iterative diffusion sampling, limiting real-time use.
• Predicted panorama resolution is limited by data and training scale.
• Slightly higher si-RMSE than NeuS+Mitsuba on real foreground objects and color shifts in some NeRF scenes.

Experimental setup

Datasets: Objaverse MatSynth Poly Haven HDRIs HDR Maps HDRI Skies DOSCH DESIGN WEB360 Laval Indoor Laval Outdoor Cube++ Objects with Lighting NeRF synthetic

Benchmarks: Laval Indoor Laval Outdoor Poly Haven PolyHaven-Peak WEB360 Cube++ Objects with Lighting Array-of-spheres (Laval Indoor)

Metrics: scale-invariant RMSE angular error normalized RMSE peak angular error RMSE SSIM user preference

Baselines: DiffusionLight StyleLight NLFE Hold-Geoffroy et al. EMLight Weber et al. EverLight DiPIR

Builds on: CogVideoX DiffusionLight LoRA DDPM

Closest papers by meaning

• DiffusionLight: Light Probes for Free by Painting a Chrome Ball (CVPR 2024 · similarity 0.89)
• UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting (NeurIPS 2025 · similarity 0.89)
• Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset (CVPR 2025 · similarity 0.89)
• Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models (CVPR 2025 · similarity 0.87)
• IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation (NeurIPS 2025 · similarity 0.87)
• MetaGS: A Meta-Learned Gaussian-Phong Model for Out-of-Distribution 3D Scene Relighting (NeurIPS 2025 · similarity 0.86)