Research map / Label-efficient and robust learning / Self-supervised and contrastive
SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SwitchLight, a physics-guided portrait relighting architecture using Cook-Torrance specular reflectance, plus a self-supervised Multi-Masked Autoencoder (MMAE) pre-training scheme, trained on a self-built light-stage OLAT dataset.
Problem
Human portrait relighting requires accurate intrinsic decomposition and rendering, but high-quality light-stage training data is scarce and expensive.
Approach
Replace the Phong specular model of Total Relight with the Cook-Torrance microfacet BRDF inside a decompose-then-render network, add a neural refinement render, and adapt masked autoencoder pre-training to UNets with dynamic multi-type masks and perceptual/adversarial losses so unlabeled data can be used before fine-tuning on relighting ground truth.
Key results
- On the OLAT test set, SwitchLight with MMAE reaches MAE 0.0933, MSE 0.0235, SSIM 0.9051, LPIPS 2.059, versus TR at 0.1643/0.0658/0.8465/3.425.
- MMAE pre-training improves over MAE pre-training (MAE 0.0952 / LPIPS 2.096 vs 0.0933 / 2.059) and the diffuse-render prediction improves over direct albedo prediction (albedo-only ablation MAE 0.1053 vs 0.1023 for diffuse render).
- In a 47-worker user study, SwitchLight was preferred over Lumos and TR on consistent lighting (0.7671 vs 0.0478 and 0.1852), facial details (0.5376 vs 0.2022 and 0.2602), and identity (0.5819 vs 0.1741 and 0.2440).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SIPR with SwitchLight | human portrait relighting · OLAT test set | MAE: 0.1715 -> 0.1023 | Helped |
| Replace TR with SwitchLight | human portrait relighting · OLAT test set | SSIM: 0.8465 -> 0.9002 | Helped |
| Add MMAE pre-training (ablation) | SwitchLight · OLAT test set | MAE: 0.1023 -> 0.0933 | Helped |
| Add MMAE pre-training (ablation) | SwitchLight · OLAT test set | SSIM: 0.9007 -> 0.9051 | Helped |
| Use MAE rather than MMAE pre-training (ablation) | SwitchLight · OLAT test set | LPIPS: 2.059 -> 2.096 | Hurt |
| Predict diffuse render instead of albedo directly (ablation) | DiffuseNet · OLAT test set | MAE: 0.1053 -> 0.1023 | Helped |
| Predict diffuse render instead of albedo directly (ablation) | DiffuseNet · OLAT test set | SSIM: 0.8985 -> 0.9002 | Helped |
| Use SwitchLight rather than TR | portrait relighting · FFHQ test set | Consistent Lighting: 0.1852 -> 0.7671 | Helped |
| Use SwitchLight rather than TR | portrait relighting · FFHQ test set | Facial Details: 0.2602 -> 0.5376 | Helped |
| Use SwitchLight rather than TR | portrait relighting · FFHQ test set | Similar Identity: 0.2440 -> 0.5819 | Helped |
Limitations
- Struggles to remove strong shadows, leading to residual shadow artifacts in albedo and relit images.
- Misinterprets reflective surfaces such as sunglasses as opaque, failing to remove highlights.
- Inaccurate albedo prediction for face paint.
Experimental setup
Datasets: OLAT (self-constructed light-stage dataset) Polyhaven HDRIs HDRMAPS HDRIs iHDRI FFHQ test set (user study) Pexels images (qualitative)
Benchmarks: OLAT test set (35 subjects, 11 lighting environments) FFHQ-test user study Pexels qualitative
Metrics: MAE MSE SSIM LPIPS
Baselines: SIPR Total Relight (TR) Lumos MAE pre-training
Builds on: Total Relight masked autoencoder (MAE) Cook-Torrance BRDF PatchGAN
Closest papers by meaning
- SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces (CVPR 2025 · similarity 0.88)
- Comprehensive Relighting: Generalizable and Consistent Monocular Human Relighting and Harmonization (CVPR 2025 · similarity 0.87)
- Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset (CVPR 2025 · similarity 0.87)
- DreamLight: Towards Harmonious and Consistent Image Relighting (NeurIPS 2025 · similarity 0.86)
- UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback (NeurIPS 2025 · similarity 0.86)
- Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.