Research map / Image generation / Image restoration
Relightful Harmonization: Lighting-aware Portrait Background Replacement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Relightful Harmonization, a lighting-aware diffusion model that harmonizes portrait foregrounds to arbitrary background images by encoding background lighting, aligning it with environment-map features, and fine-tuning on synthetic real-image pairs.
Problem
Portrait harmonization for background replacement typically adjusts only global color and brightness, ignoring lighting direction and shadows from the background; relighting methods need HDR environment maps that are not available in casual photography.
Approach
A three-stage training of an InstructPix2Pix-initialized diffusion model with a background-derived lighting feature branch, an alignment network mapping background lighting features to environment-map features, and fine-tuning on a synthetic dataset generated from real natural images by the model itself, so no environment map is needed at inference.
Key results
- On the light stage test set, the method reaches PSNR 20.527 and LPIPS 0.159, versus 19.304 PSNR and 0.338 LPIPS for the best prior harmonization baseline (Harmonizer).
- On the natural image test set, PSNR 23.562 and LPIPS 0.097 versus 21.566 PSNR and 0.113 LPIPS for INR.
- Amazon Mechanical Turk pairwise study (1750 ratings) prefers the method over PIH, INR, PCT and Harmonizer at rates of 0.713, 0.702, 0.845 and 0.639.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the strongest baseline, Harmonizer, with Relightful Harmonization | Portrait harmonization · Light stage test set | MSE: 0.015 -> 0.012 | Helped |
| replace the strongest baseline, Harmonizer, with Relightful Harmonization | Portrait harmonization · Light stage test set | PSNR: 19.304 -> 20.527 | Helped |
| replace the strongest baseline, INR, with Relightful Harmonization | Portrait harmonization · Light stage test set | SSIM: 0.823 -> 0.848 | Helped |
| replace the strongest baseline, PIH, with Relightful Harmonization | Portrait harmonization · Light stage test set | LPIPS: 0.330 -> 0.159 | Helped |
| replace the strongest baseline, INR, with Relightful Harmonization | Portrait harmonization · Natural image test set | MSE: 0.009 -> 0.005 | Helped |
| replace the strongest baseline, INR, with Relightful Harmonization | Portrait harmonization · Natural image test set | PSNR: 21.566 -> 23.562 | Helped |
| replace the strongest baseline, Harmonizer, with Relightful Harmonization | Portrait harmonization · Natural image test set | SSIM: 0.905 -> 0.913 | Helped |
| replace the strongest baseline, Harmonizer, with Relightful Harmonization | Portrait harmonization · Natural image test set | LPIPS: 0.108 -> 0.097 | Helped |
| add background-image lighting conditioning (ablation) | Relightful Harmonization · Light stage test set | PSNR: 18.768 -> 19.748 | Helped |
| use environment-map lighting conditioning instead of background-image conditioning (ablation) | Relightful Harmonization · Light stage test set | PSNR: 19.748 -> 21.626 | Helped |
| add lighting representation alignment to background-conditioned lighting (ablation) | Relightful Harmonization · Light stage test set | PSNR: 19.748 -> 20.439 | Helped |
| add photorealism finetuning on synthesized natural-image pairs (ablation) | Relightful Harmonization · Natural image test set | PSNR: 20.006 -> 23.562 | Helped |
Surprising findings
- Background-only lighting conditioning (Model#1) does not improve natural-image metrics over the baseline, while the synthetic fine-tuning stage drives most of the natural-image gain (MSE 0.005 vs 0.012).
Limitations
- Output resolution capped at 512x512, which can hurt facial detail on small faces.
- Subtle variations in clothing and skin tone may occur.
- Training relies on a proprietary light stage dataset with limited subjects (100).
Experimental setup
Datasets: light stage rendered dataset natural images with synthetic pairs
Benchmarks: light stage test set (500 pairs) natural image synthetic test set (200 pairs) real-world portraits
Metrics: MSE PSNR SSIM LPIPS user preference rate
Baselines: INR PCT Harmonizer PIH TR (Total Relighting) Transformer relighting baseline
Builds on: InstructPix2Pix Total Relighting PIH Harmonizer
Closest papers by meaning
- DreamLight: Towards Harmonious and Consistent Image Relighting (NeurIPS 2025 · similarity 0.91)
- Comprehensive Relighting: Generalizable and Consistent Monocular Human Relighting and Harmonization (CVPR 2025 · similarity 0.89)
- SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces (CVPR 2025 · similarity 0.89)
- Text2Relight: Creative Portrait Relighting with Text Guidance (AAAI 2025 · similarity 0.88)
- Progressive Painterly Image Harmonization from Low-Level Styles to High-Level Styles (AAAI 2024 · similarity 0.88)
- Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.