Research map / Video / Video generation
UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes a Cosmos DiT video diffusion model to jointly denoise albedo and relit video in one pass, trained on synthetic multi-illumination data, MIT multi-illumination, and auto-labeled real videos.
Problem
Single image or video relighting is hard because paired multi-illumination data is scarce, and two-stage inverse-then-forward rendering pipelines suffer error accumulation and fail on complex materials and lighting.
Approach
Jointly denoise albedo and relit video as concatenated latent tokens along the temporal dimension in a video DiT, with type embeddings and condition masks, so demodulation acts as an implicit prior for relighting; real videos are auto-labeled with albedo to add training diversity.
Key results
- On MIT multi-illumination relighting, Ours reaches PSNR 20.76 / SSIM 0.749 versus 17.29 / 0.622 for DiffusionRenderer (Cosmos).
- On SyntheticScenes relighting, Ours achieves PSNR 26.97 and LPIPS 0.190, best among compared methods, with DiffusionRenderer (Cosmos) at 26.61 PSNR.
- Single-pass joint inference takes 445.5 s per 57-frame video on one A100, versus 566.6 s for DiffusionRenderer and 780.0 s for its Cosmos variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use joint albedo and relighting modeling instead of direct relighting (ablation) | UniRelight · SyntheticScenes | PSNR: 26.42 -> 26.97 | Helped |
| use joint albedo and relighting modeling instead of direct relighting (ablation) | UniRelight · SyntheticScenes | SSIM: 0.842 -> 0.847 | Helped |
| use joint albedo and relighting modeling instead of direct relighting (ablation) | UniRelight · SyntheticScenes | LPIPS: 0.191 -> 0.190 | Helped |
| condition on ground-truth albedo (ablation) | UniRelight joint model · SyntheticScenes | PSNR: 26.97 -> 27.15 | Helped |
| condition on ground-truth albedo (ablation) | UniRelight joint model · SyntheticScenes | SSIM: 0.847 -> 0.857 | Helped |
| condition on ground-truth albedo (ablation) | UniRelight joint model · SyntheticScenes | LPIPS: 0.190 -> 0.181 | Helped |
| add real-world auto-labeled data (ablation) | UniRelight · StreetScenes | User preference: $45\% \pm \phantom{0}8\%$ | Helped |
| remove joint modeling (ablation) | UniRelight without auto-labeled data · StreetScenes | User preference for joint model: $68\% \pm 14\%$ | Helped |
| use estimated albedo at inference time (ablation) | UniRelight without auto-labeled data · StreetScenes | User preference for base model: $54\% \pm 13\%$ | No clear effect |
| use UniRelight instead of DiffusionRenderer (Cosmos) | Relighting · SyntheticScenes | PSNR: 26.61 -> 26.97 | Helped |
| use UniRelight instead of Neural Gaffer | Relighting · MIT multi-illumination | PSNR: 17.87 -> 20.76 | Helped |
| use UniRelight instead of DiffusionRenderer (Cosmos) | Albedo estimation · SyntheticScenes | PSNR: 28.56 -> 28.07 | Hurt |
Surprising findings
- On MIT multi-illumination, Neural Gaffer obtains the lowest LPIPS (0.241) despite making minimal changes to the input, which the authors attribute to small light direction deviations.
- Albedo estimation of the proposed model is slightly below DiffusionRenderer (Cosmos) on SyntheticScenes (PSNR 28.07 vs 28.56), and the joint model's relighting gain over the pure relighting ablation is small (26.97 vs 26.42 PSNR).
Limitations
- Cannot handle emitting objects such as toggled lights within the scene.
- No text-based relighting; lighting is specified only by environment maps.
- Auto-labeled real-world data is lower quality and sparsely labeled, and real-world albedo labels come from a re-implemented inverse renderer.
Experimental setup
Datasets: SyntheticScenes-style synthetic renders (Objaverse LVIS, PolyHaven HDRI, PBR materials) MIT multi-illumination auto-labeled real-world internet videos
Benchmarks: SyntheticScenes MIT multi-illumination StreetScenes IC-Light comparison set
Metrics: PSNR SSIM LPIPS user preference
Baselines: DiLightNet Neural Gaffer DiffusionRenderer DiffusionRenderer (Cosmos) IntrinsicImageDiffusion IC-Light
Builds on: VideoJAM DiffusionRenderer Cosmos-Predict1 EDM RGB↔X
Closest papers by meaning
- UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback (NeurIPS 2025 · similarity 0.93)
- Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset (CVPR 2025 · similarity 0.90)
- SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces (CVPR 2025 · similarity 0.90)
- Neural Gaffer: Relighting Any Object via Diffusion (NeurIPS 2024 · similarity 0.90)
- ReCap: Better Gaussian Relighting with Cross-Environment Captures (CVPR 2025 · similarity 0.90)
- SyncLight: Single-Edit Multi-View Relighting (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.