Research map / Video / Video generation
Comprehensive Relighting: Generalizable and Consistent Monocular Human Relighting and Harmonization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Comprehensive Relighting, a diffusion-based all-in-one framework that relights and harmonizes monocular human images and videos using SH coarse shading plus background conditioning, with an unsupervised cycle-consistency temporal lighting module added at inference.
Problem
Monocular human relighting is limited to specific scenarios (faces or static humans) due to scarce ground-truth data, and existing methods handle only one lighting control source and lack temporal coherence for video.
Approach
Repurpose a pretrained latent diffusion model conditioned on a pixel-aligned coarse SH shading map and a background image (coarse-to-fine residual relighting), plus an unsupervised temporal lighting module trained with lighting cycle consistency on real videos and combined at inference via spatio-temporal feature blending and guided residual refinement.
Key results
- On synthetic video Scenario 3, the method reaches 26.61 PSNR / 0.94 SSIM versus the best baseline GFR at 25.49 / 0.93 (fidelity), and 38.32 tPSNR-related temporal score vs 37.53 for RHW.
- On static human testing, the full model gets L1 0.01012, PSNR 28.778, SSIM 0.949 in the ablation, and full-body PSNR 30.81 vs 29.51 for GFR.
- Temporal module with blending lowers tL1 from 6.552 to 4.019 (x10^-3) and raises tPSNR from 31.03 to 33.59 on synthetic sequences.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use coarse-to-fine relighting instead of direct end-to-end lighting conditioning (ablation) | Comprehensive Relighting · Coarse-to-fine relighting ablation | PSNR: 26.418 -> 28.419 | Helped |
| add diffusion-model fine-grained relighting to pixel-aligned coarse shading (ablation) | Comprehensive Relighting · Coarse-to-fine relighting ablation | PSNR: 17.099 -> 28.419 | Helped |
| add background-image conditioning (ablation) | Comprehensive Relighting · Coarse-to-fine relighting ablation | PSNR: 27.346 -> 28.419 | Helped |
| add guided refinement (ablation) | Comprehensive Relighting · Coarse-to-fine relighting ablation | PSNR: 28.419 -> 28.778 | Helped |
| add temporal lighting module (ablation) | Comprehensive Relighting · Synthetic sequences | tPSNR: 31.028 -> 32.266 | Helped |
| add recurrent feature blending to the temporal lighting module (ablation) | Comprehensive Relighting · Synthetic sequences | tPSNR: 32.266 -> 33.588 | Helped |
Limitations
- Relighting diffusion model is computationally heavy.
- Noise in detected masks and surface normals degrades temporal coherence.
- Evaluated mostly on synthetic data; no public ground truth for dynamic-human video relighting.
Experimental setup
Datasets: synthetic human renderings (50K) LightStage OLAT captures (50K) real videos from existing works (50K frames) DeepFashion Pexels
Benchmarks: internal synthetic static human test set synthetic video scenarios 1-3 DeepFashion Pexels
Metrics: PSNR SSIM LPIPS tPSNR tSSIM tLPIPS L1 tL1
Baselines: DPR RHW GFR LPBR
Builds on: Stable Diffusion ControlNet Total Relighting LPBR TokenFlow
Closest papers by meaning
- DreamLight: Towards Harmonious and Consistent Image Relighting (NeurIPS 2025 · similarity 0.90)
- Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset (CVPR 2025 · similarity 0.90)
- Relightful Harmonization: Lighting-aware Portrait Background Replacement (CVPR 2024 · similarity 0.89)
- UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting (NeurIPS 2025 · similarity 0.89)
- SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces (CVPR 2025 · similarity 0.89)
- UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.