Research map / Video / Video generation
Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Lux Post Facto fine-tunes Stable Video Diffusion for HDR-map-conditioned portrait video relighting, trained on a hybrid of static OLAT images and in-the-wild talking-head videos via a reference-based appearance copy auxiliary task.
Problem
Portrait video relighting must be both photorealistic and temporally stable, but existing methods trade off spatial quality for temporal consistency or require scarce paired dynamic OLAT video data.
Approach
Condition a pretrained video diffusion model (SVD) on HDR maps encoded as per-direction light embeddings injected via cross-attention, and train on a hybrid dataset (OLAT-rendered images with simulated camera motion plus in-the-wild videos with pseudo albedo) using reference-based appearance copy as an auxiliary task to gain temporal consistency without paired lit video.
Key results
- On the held-out OLAT image test set, Lux Post Facto reaches LPIPS 0.1158, PSNR 24.62 and SSIM 0.8278, beating Total Relighting (LPIPS 0.1794, PSNR 22.44) and SwitchLight (LPIPS 0.2129, PSNR 19.87).
- Replacing the CLIP image encoder with the proposed HDR light embedding improves PSNR from 20.73 to 24.62 in the lighting-control ablation.
- Video comparisons against NVPR and SwitchLight are qualitative only, since no ground-truth relit videos exist.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Lux Post Facto instead of Total Relighting | portrait image relighting · Left-out OLAT test set | LPIPS: 0.1794 -> 0.1158 | Helped |
| use Lux Post Facto instead of Total Relighting | portrait image relighting · Left-out OLAT test set | NIQE: 6.458 -> 5.653 | Helped |
| use Lux Post Facto instead of Total Relighting | portrait image relighting · Left-out OLAT test set | PSNR: 22.44 -> 24.62 | Helped |
| use Lux Post Facto instead of Total Relighting | portrait image relighting · Left-out OLAT test set | SSIM: 0.7793 -> 0.8278 | Helped |
| replace light-token conditioning with CLIP image conditioning (ablation) | Lux Post Facto · Left-out OLAT test set | LPIPS: 0.1158 -> 0.1747 | Hurt |
| replace light-token conditioning with CLIP image conditioning (ablation) | Lux Post Facto · Left-out OLAT test set | NIQE: 5.653 -> 5.831 | Hurt |
| replace light-token conditioning with CLIP image conditioning (ablation) | Lux Post Facto · Left-out OLAT test set | PSNR: 24.62 -> 20.73 | Hurt |
| replace light-token conditioning with CLIP image conditioning (ablation) | Lux Post Facto · Left-out OLAT test set | SSIM: 0.8278 -> 0.7808 | Hurt |
| add motion-rich in-the-wild video training to the OLAT-only training (ablation) | Lux Post Facto · In-the-wild portrait video relighting | temporal consistency | Helped |
| add motion-rich in-the-wild video training to the OLAT-only training (ablation) | Lux Post Facto · In-the-wild portrait video relighting | sharpness | Helped |
Surprising findings
- Training only on the lighting-rich OLAT data with simulated camera motion yields blurry results on moving subjects, so the in-the-wild motion data is needed.
Limitations
- Video comparisons against baselines are qualitative only because no ground-truth relit video exists.
- The delighting model relies on pseudo albedo from a frame-wise image delighting model.
- Long-sequence inference is autoregressive, with a fixed 30-frame window and 4 overlapping frames.
Experimental setup
Datasets: VFHQ Light stage OLAT dataset (in-house, 67 subjects) PolyHeaven HDR maps
Benchmarks: in-house OLAT test set (7 held-out subjects) in-the-wild portrait videos and images
Metrics: PSNR SSIM LPIPS NIQE
Baselines: NVPR SwitchLight Total Relighting CLIP image encoder conditioning
Builds on: Stable Video Diffusion Total Relighting VFHQ
Closest papers by meaning
- SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces (CVPR 2025 · similarity 0.91)
- UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting (NeurIPS 2025 · similarity 0.90)
- Comprehensive Relighting: Generalizable and Consistent Monocular Human Relighting and Harmonization (CVPR 2025 · similarity 0.90)
- High-Fidelity Relightable Monocular Portrait Animation with Lighting-Controllable Video Diffusion Model (CVPR 2025 · similarity 0.89)
- UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback (NeurIPS 2025 · similarity 0.89)
- Real-time 3D-aware Portrait Video Relighting (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.