Research map / Image generation / Editing and style transfer
Text2Relight: Creative Portrait Relighting with Text Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Text2Relight builds a synthetic text-lighting-portrait relighting dataset (1.5M pairs) via LLM-generated hierarchical prompts, text-guided lighting generation, and image-based relighting, then fine-tunes InstructPix2Pix with auxiliary shadow-removal and light-positioning tasks for text-guided portrait relighting.
Problem
Text-guided portrait relighting is hard because no datasets pair lighting-descriptive text with identity-preserving relit images, and general text-guided editors distort content and fail on lighting prompts.
Approach
Synthesize large-scale relighting pairs by generating diverse lighting text via a categorical hierarchy with ChatGPT, generating lighting images from text, and transferring lighting to foreground (single-image or OLAT) and background (point-light reconstruction), then train a diffusion editor with auxiliary tasks.
Key results
- Ours achieves SSIM 0.546, LPIPS 0.401, CLIP vision score 0.886 and face identity score 0.584, outperforming IP2P, GLIDE, and MGIE.
- In user study, 66.17% of participants preferred Text2Relight results versus 14.83% for IP2P and 15.50% for MGIE.
- Removing the mask condition or auxiliary task data lowers the full model's SSIM from 0.551 to 0.535 and 0.545 respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Text2Relight instead of MGIE | Text-guided portrait relighting · Text-guided portrait relighting | SSIM: 0.415 -> 0.546 | Helped |
| Use Text2Relight instead of MGIE | Text-guided portrait relighting · Text-guided portrait relighting | LPIPS: 0.464 -> 0.401 | Helped |
| Use Text2Relight instead of MGIE | Text-guided portrait relighting · Text-guided portrait relighting | CVS: 0.802 -> 0.886 | Helped |
| Use Text2Relight instead of MGIE | Text-guided portrait relighting · Text-guided portrait relighting | FIS: 0.465 -> 0.584 | Helped |
| Use Text2Relight instead of MGIE | Text-guided portrait relighting · Text-guided portrait relighting | LS: 3.3 -> 3.8 | Helped |
| Add auxiliary task data (ablation) | Text2Relight · Text-guided portrait relighting | SSIM: 0.545 -> 0.551 | Helped |
| Add auxiliary task data (ablation) | Text2Relight · Text-guided portrait relighting | LPIPS: 0.419 -> 0.399 | Helped |
| Add auxiliary task data (ablation) | Text2Relight · Text-guided portrait relighting | CVS: 0.866 -> 0.882 | Helped |
| Add mask conditioning (ablation) | Text2Relight · Text-guided portrait relighting | SSIM: 0.535 -> 0.551 | Helped |
| Add mask conditioning (ablation) | Text2Relight · Text-guided portrait relighting | LPIPS: 0.423 -> 0.399 | Helped |
| Add mask conditioning (ablation) | Text2Relight · Text-guided portrait relighting | CVS: 0.865 -> 0.882 | Helped |
| Use Text2Relight instead of MGIE | Text-guided portrait relighting · User preference study | preference: 15.50% -> 66.17% | Helped |
Surprising findings
- General text-guided editors mostly perform style transfer rather than actual relighting, per the authors' qualitative analysis.
- GLIDE obtains a face identity score of -0.969, indicating it largely replaces the human subject.
Limitations
- Model sometimes generates strong unnatural point lights in the background.
- Weak spatial lighting context in text makes localizing text-specified lights difficult.
Experimental setup
Datasets: lightstage OLAT portrait data real-world portrait images 20k real world images HDR panorama maps synthesized relighting pairs
Benchmarks: synthesized test set from the data pipeline real-world portrait images
Metrics: SSIM LPIPS CLIP vision score face identity score LLaVA lighting adjustment score user preference
Baselines: InstructPix2Pix GLIDE MGIE
Builds on: InstructPix2Pix latent consistency model Stable Diffusion LLaVA
Closest papers by meaning
- DreamLight: Towards Harmonious and Consistent Image Relighting (NeurIPS 2025 · similarity 0.90)
- SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces (CVPR 2025 · similarity 0.89)
- Relightful Harmonization: Lighting-aware Portrait Background Replacement (CVPR 2024 · similarity 0.88)
- Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset (CVPR 2025 · similarity 0.87)
- Comprehensive Relighting: Generalizable and Consistent Monocular Human Relighting and Harmonization (CVPR 2025 · similarity 0.87)
- TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.