Research map / Image generation / Autoregressive and latent generation
UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniLumos, a flow-matching (Wan2.1-based) unified image and video relighting model supervised with depth/normal feedback computed on decoded RGB outputs and path-consistency learning for few-step inference, plus the LumosData pipeline and LumosBench attribute-level benchmark.
Problem
Diffusion-based relighting operates in semantic latent space, so outputs often have misaligned shadows, overexposed highlights, and wrong light directions; video methods are slow (multi-pass) and lack explicit physical supervision.
Approach
Supervise the flow-matching relighting model in RGB space with depth and surface-normal losses from a frozen dense estimator (Lotus), combined with path-consistency learning for few-step generation, trained on LumosData with structured six-dimension lighting annotations.
Key results
- Image relighting: UniLumos reaches PSNR 26.719 and Lumos avg score 0.912 versus 25.572 and 0.791 for SynthLight.
- Video relighting: UniLumos reaches PSNR 25.031 and R-Motion 1.436 versus 20.784 and 1.582 for Light-A-Video + Wan2.1.
- Paper claims a 20x speedup over prior video relighting methods for 49-frame 480p videos.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add depth feedback (ablation) | UniLumos · Video Relighting (internal test) | Dense L2 Error: 0.265 -> 0.147 | Helped |
| add normal feedback (ablation) | UniLumos · Video Relighting (internal test) | PSNR: 22.115 -> 25.031 | Helped |
| add depth and normal feedback (ablation) | UniLumos · Video Relighting (internal test) | Dense L2 Error: 0.297 -> 0.147 | Helped |
| add path consistency learning (ablation) | UniLumos · Video Relighting (internal test) | Dense L2 Error: 0.153 -> 0.147 | Helped |
| train on both images and videos rather than only videos (ablation) | UniLumos · Video Relighting (internal test) | PSNR: 22.487 -> 25.031 | Helped |
| train on both images and videos rather than only images (ablation) | UniLumos · Video Relighting (internal test) | R-Motion: 2.429 -> 1.436 | Helped |
| use structured Lumos captions instead of vanilla scene-level captions (ablation) | UniLumos · LumosBench | Avg. Score: 0.662 -> 0.773 | Helped |
| use UniLumos rather than Light-A-Video + Wan2.1 | Video Relighting · Video Relighting (internal test) | PSNR: 20.784 -> 25.031 | Helped |
| use UniLumos rather than SynthLight | Image Relighting · Image Relighting (internal test) | Avg. Score: 0.791 -> 0.912 | Helped |
Surprising findings
- Removing path consistency gives slightly higher PSNR (25.317 vs 25.031) and SSIM in the ablation, while the authors argue it mainly buys efficiency.
- Training only on images gives a higher PSNR than the unified model on video (24.471 vs 25.031 is lower, but Only Image R-Motion is 2.429, worse than the video baselines).
Limitations
- Does not produce physically quantifiable lighting outputs such as radiance or illuminance.
- Evaluation relies on an internal dataset and a VLM judge (Qwen2.5-VL) for lighting attributes.
- Depth/normal supervision depends on a frozen pseudo-label estimator (Lotus) and pseudo-ground-truth from the reference input.
Experimental setup
Datasets: Panda70M LumosData (IC-Light relit images and videos) StanfordORB Navi
Benchmarks: internal test set StanfordORB Navi LumosBench
Metrics: PSNR SSIM LPIPS R-Motion Lumos Score Dense L2 Error
Baselines: SwitchLight DiLightNet IC-Light SynthLight Light-A-Video IC-Light Per Frame
Builds on: Wan2.1 flow matching path consistency IC-Light Lotus
Closest papers by meaning
- UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting (NeurIPS 2025 · similarity 0.93)
- Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset (CVPR 2025 · similarity 0.89)
- SyncLight: Single-Edit Multi-View Relighting (NeurIPS 2026 · similarity 0.89)
- MetaGS: A Meta-Learned Gaussian-Phong Model for Out-of-Distribution 3D Scene Relighting (NeurIPS 2025 · similarity 0.89)
- Comprehensive Relighting: Generalizable and Consistent Monocular Human Relighting and Harmonization (CVPR 2025 · similarity 0.89)
- DreamLight: Towards Harmonious and Consistent Image Relighting (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.