Research map / Image generation / Editing and style transfer
SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds SynthLight, a Stable Diffusion-based portrait relighting model trained on ~1.26M Blender-rendered synthetic faces with multi-task text-to-portrait training and classifier-free-guidance inference adaptation.
Problem
Portrait relighting from a single image under a target environment map, where lighting-labeled training data is scarce (light stage) and synthetic renders have a domain gap to real photographs.
Approach
Frame relighting as learned re-rendering: train a latent diffusion model on physically based synthetic head renders paired across environment maps, then bridge the synthetic-to-real gap via joint text-to-portrait training on real LAION portraits and an image-conditioned classifier-free guidance scheme at inference that preserves input detail.
Key results
- On the held-out synthetic test set, SynthLight reaches PSNR 29.57 and LPIPS 0.063, outperforming IC-Light, SwitchLight, Neural Gaffer and DiLightNet.
- On the light stage test set, SynthLight (PSNR 19.70, LPIPS 0.165) is comparable to SwitchLight (PSNR 20.30, LPIPS 0.141), a method trained on light stage data.
- In a user study with 482 responses, SynthLight was preferred over IC-Light for lighting 92% of the time and over SwitchLight for identity 70% of the time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SynthLight instead of SwitchLight | Portrait relighting · Test Synthetic | LPIPS: 0.088 -> 0.063 | Helped |
| Use SynthLight instead of SwitchLight | Portrait relighting · Test Light Stage | LPIPS: 0.141 -> 0.165 | Hurt |
| Add multi-task training to the Base configuration (ablation) | SynthLight · Test Synthetic | LPIPS: 0.066 -> 0.066 | No clear effect |
| Add multi-task training to the Base configuration (ablation) | SynthLight · Test Light Stage | LPIPS: 0.210 -> 0.186 | Helped |
| Add inference adaptation to the Base configuration (ablation) | SynthLight · Test Synthetic | LPIPS: 0.066 -> 0.062 | Helped |
| Add inference adaptation to the Base configuration (ablation) | SynthLight · Test Light Stage | LPIPS: 0.210 -> 0.178 | Helped |
| Add light stage data during training (ablation) | SynthLight · Test Light Stage | LPIPS: 0.165 -> 0.156 | Helped |
| Add light stage data during training (ablation) | SynthLight · Test Synthetic | LPIPS: 0.063 -> 0.065 | Hurt |
| Initialize with Stable Diffusion 1.5 instead of IC-Light (ablation) | SynthLight · Test Synthetic | LPIPS: 0.063 -> 0.069 | Hurt |
| Initialize with Stable Diffusion 1.5 instead of IC-Light (ablation) | SynthLight · Test Light Stage | LPIPS: 0.165 -> 0.177 | Hurt |
| Condition lighting on the environment map rather than background (ablation) | SynthLight · Portrait relighting | lighting consistency | Helped |
Surprising findings
- Adding real light stage data to training fixed identity and texture issues but failed to remove strong input shadows, which the authors attribute to rarity of strong shadows in light stage captures.
Limitations
- Strong input shadows persist in relit output because the model cannot remove lighting baked into the input portrait.
- Oversaturation under strong sunlight appears in outputs.
- Trained only on simple headshots without accessories, facial hair, or hats.
Experimental setup
Datasets: synthetic Blender head renders (Cycles) LAION human portraits light stage OLAT data (ablation)
Benchmarks: Test Synthetic (held-out synthetic faces) Test Light Stage (OLAT-rendered pairs)
Metrics: LPIPS SSIM PSNR FaceNet distance user preference rate
Baselines: IC-Light SwitchLight Neural Gaffer DiLightNet Total Relighting
Builds on: Stable Diffusion Palette IC-Light InstructPix2Pix Neural Gaffer
Closest papers by meaning
- Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset (CVPR 2025 · similarity 0.91)
- Neural Gaffer: Relighting Any Object via Diffusion (NeurIPS 2024 · similarity 0.90)
- UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting (NeurIPS 2025 · similarity 0.90)
- Comprehensive Relighting: Generalizable and Consistent Monocular Human Relighting and Harmonization (CVPR 2025 · similarity 0.89)
- DreamLight: Towards Harmonious and Consistent Image Relighting (NeurIPS 2025 · similarity 0.89)
- Text2Relight: Creative Portrait Relighting with Text Guidance (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.