Research map / Video / Video generation
IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IllumiCraft, a DiT-based video diffusion framework that conditions on HDR environment maps, relit foreground videos (with optional background image), and 3D point-track videos to perform controllable video relighting, and releases a 20,170-pair dataset built with the IllumiPipe pipeline.
Problem
Video relighting with explicit lighting control lacks geometric guidance, so existing video relighting methods lose lighting fidelity and temporal coherence when scene geometry changes.
Approach
Jointly condition a Wan2.1 DiT on illumination (learnable tokens from an HDR-map encoder, with dropout), appearance (relit foreground and zero-padded background reference latents channel-concatenated), and geometry (3D point-track videos injected via cloned DiT blocks with zero-initialized linear layers, ControlNet-style).
Key results
- Text-conditioned: FVD 2186.40 vs. 3857-4914 for baselines, and LPIPS 0.5623 and PSNR 12.03, the best of the four methods; the paper reports a 43% FVD reduction versus the strongest baseline.
- Background-conditioned 49-frame: FVD 1072.38, LPIPS 0.2592, PSNR 19.44, about 37% FVD reduction versus Light-A-Video.
- Ablation: adding 3D geometry guidance (I+G) reduces FVD from 1305.45 to 1072.38 (about 18%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 3D geometry guidance to illumination guidance (ablation) | IllumiCraft · Background-conditioned video relighting, 49 frames | FVD: 1305.45 -> 1072.38 | Helped |
| Add 3D geometry guidance to illumination guidance (ablation) | IllumiCraft · Background-conditioned video relighting, 49 frames | LPIPS: 0.2816 -> 0.2592 | Helped |
| Add 3D geometry guidance to illumination guidance (ablation) | IllumiCraft · Background-conditioned video relighting, 49 frames | PSNR: 18.28 -> 19.44 | Helped |
| Add 3D geometry guidance to illumination guidance (ablation) | IllumiCraft · Background-conditioned video relighting, 49 frames | Text Alignment: 0.3211 -> 0.3292 | Helped |
| Add 3D geometry guidance to illumination guidance (ablation) | IllumiCraft · Background-conditioned video relighting, 49 frames | Temporal Consistency: 0.9864 -> 0.9945 | Helped |
| Use 50% rather than 40%, 60%, or 70% HDR-feature dropout (ablation) | IllumiCraft · Text-conditioned video relighting | Text Alignment: 0.3325; 0.3342; 0.3312; 0.3301 | Mixed |
| Use 50% rather than 40%, 60%, or 70% HDR-feature dropout (ablation) | IllumiCraft · Background-conditioned video relighting | FVD: 1048.24; 1072.38; 1065.21; 1051.14 | Mixed |
| Use 30% rather than 10%, 20%, or 40% 3D-tracking dropout (ablation) | IllumiCraft · Text-conditioned video relighting | FVD: 2285.32; 2251.21; 2186.40; 2234.35 | Helped |
| Use 30% rather than 10%, 20%, or 40% 3D-tracking dropout (ablation) | IllumiCraft · Background-conditioned video relighting | FVD: 1154.32; 1102.21; 1072.38; 1098.35 | Helped |
| Drop reference images with 10% probability rather than 5%, 20%, or 30% (ablation) | IllumiCraft · Text-conditioned video relighting | Text Alignment: 0.3331; 0.3342; 0.3341; 0.3338 | Helped |
| Drop reference images with 10% probability rather than 5%, 20%, or 30% (ablation) | IllumiCraft · Background-conditioned video relighting | FVD: 1065.83; 1072.38; 1105.28; 1127.32 | Mixed |
Surprising findings
- Dropping 3D tracking videos in 30% of training samples gave better results than lower drop rates, and dropping the HDR input at 50% gave the best trade-off.
Limitations
- Fidelity depends on the generative prior of the base model, causing texture blurring in some cases.
- Failure cases include misaligned foliage and occluded light sources treated as foreground.
- Relies on DiffusionLight for HDR maps, so its lighting quality limits relit output; ethical concerns about deepfakes.
Experimental setup
Datasets: Pexels videos
Benchmarks: 50 held-out Pexels videos
Metrics: FVD LPIPS PSNR CLIP text alignment CLIP temporal consistency
Baselines: IC-Light AnyV2V Light-A-Video RelightVid
Builds on: Wan2.1 ControlNet DiffusionLight Light-A-Video IC-Light
Closest papers by meaning
- Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models (CVPR 2024 · similarity 0.89)
- UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting (NeurIPS 2025 · similarity 0.89)
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.88)
- Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset (CVPR 2025 · similarity 0.88)
- LightIt: Illumination Modeling and Control for Diffusion Models (CVPR 2024 · similarity 0.88)
- UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.