Research map / Image generation / Image restoration
PHR-DIFF: Portrait Highlights Removal via Patch-aware Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PHR-DIFF, a patch-based conditional diffusion model with patch-residual training and patch-aware cross-attention sampling for removing specular highlights from portraits.
Problem
Portrait specular highlights degrade aesthetics and downstream tasks; existing methods struggle to remove highlights fully while restoring disturbed facial regions with high fidelity.
Approach
Diffuses highlight-free portraits patch-by-patch with positional embeddings, adds a full-resolution latent encoding as a residual term to enforce global facial coherence, and at sampling uses cross-attention from non-highlight patch features (PAHR) to guide highlight removal in each patch.
Key results
- On PFSD, PHR-DIFF reaches PSNR 35.07, SSIM 0.957, LPIPS 0.092, beating the prior best Zheng et al. 2024 (33.84 PSNR, 0.945 SSIM).
- Trained only on PFSD, it generalizes to Spec-Face with PSNR 33.68 and SSIM 0.925, again best among compared methods.
- User study on 30 FFHQ cases (40 volunteers) gives PHR-DIFF the highest B-T score of 1.937 versus 1.741 for the next best method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PHR-DIFF instead of Zheng et al. 2024 | portrait highlight removal · PFSD | LPIPS: 0.097 -> 0.092 | Helped |
| use PHR-DIFF instead of Zheng et al. 2024 | portrait highlight removal · PFSD | SSIM: 0.945 -> 0.957 | Helped |
| use PHR-DIFF instead of Zheng et al. 2024 | portrait highlight removal · Spec-Face | LPIPS: 0.114 -> 0.105 | Helped |
| use PHR-DIFF instead of Zheng et al. 2024 | portrait highlight removal · Spec-Face | SSIM: 0.908 -> 0.925 | Helped |
| use PHR-DIFF instead of Zheng et al. 2024 | portrait highlight removal · FFHQ | B-T score: 1.741 -> 1.937 | Helped |
| use 32×32 rather than 128×128 patches (ablation) | PHR-DIFF · patch-size ablation | SSIM: 0.939 -> 0.959 | Helped |
| use 32×32 rather than 128×128 patches (ablation) | PHR-DIFF · patch-size ablation | LPIPS: 0.107 -> 0.089 | Helped |
| use smaller patches (ablation) | PHR-DIFF · patch-size ablation | sampling time: 1.7 -> 4.1 | Hurt |
| add patchify training (ablation) | PHR-DIFF · PHR-DIFF ablation | SSIM: 0.919 -> 0.929 | Helped |
| add patch-residual training (ablation) | PHR-DIFF · PHR-DIFF ablation | SSIM: 0.929 -> 0.935 | Helped |
| add patch-residual training (ablation) | PHR-DIFF · PHR-DIFF ablation | LPIPS: 0.123 -> 0.110 | Helped |
| add patch-wise sampling (ablation) | PHR-DIFF · PHR-DIFF ablation | SSIM: 0.935 -> 0.944 | Helped |
Surprising findings
- Smaller patch sizes improve quality (32x32 best SSIM/LPIPS) but at higher sampling time, so 64x64 was chosen as a tradeoff.
Limitations
- Fails when highlights strongly obscure high-frequency information in highlight regions.
- Suboptimal when facial skin tone is very dark or very bright, since non-highlight guidance is limited.
- Highlight masks depend on a separate pre-trained detection network.
Experimental setup
Datasets: PFSD
Benchmarks: PFSD Spec-Face FFHQ CelebA
Metrics: LPIPS DreamSim SSIM MS-SSIM PSNR B-T score (user study)
Baselines: Liang et al. 2021 Muhammad et al. 2020 Wu et al. 2021 Wu et al. 2023 Zheng et al. 2024
Builds on: DDPM DDIM Saharia et al. 2022 U-Net denoiser Guo et al. 2023a shadow removal diffusion
Closest papers by meaning
- Self-Supervised Selective-Guided Diffusion Model for Old-Photo Face Restoration (NeurIPS 2025 · similarity 0.86)
- Diff-Shadow: Global-guided Diffusion Model for Shadow Removal (AAAI 2025 · similarity 0.86)
- DiffRetouch: Using Diffusion to Retouch on the Shoulder of Experts (AAAI 2025 · similarity 0.86)
- Detail-Preserving Latent Diffusion for Stable Shadow Removal (CVPR 2025 · similarity 0.85)
- SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces (CVPR 2025 · similarity 0.85)
- Hierarchical Adaptive Filtering Network for Text Image Specular Highlight Removal (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.