Research map / Image generation / Editing and style transfer
PFStorer: Personalized Face Restoration and Super-Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Personalizes a diffusion-based face restoration model (initialized from StableSR) with a few reference images using adapter blocks, a learnable gating vector, and a generative regularizer to restore faithful identity.
Problem
Face restoration from low-quality inputs is ill-posed, so generic restoration models often produce realistic but identity-unfaithful faces; the paper targets identity-faithful restoration and super-resolution.
Approach
Personalize a base restoration diffusion model via independent trainable personalization blocks (cross-attention to reference-image features) gated by learnable per-layer vectors that keep the base priors frozen, plus a generative regularizer that drops the LQ condition to prevent reliance on the input's identity cues.
Key results
- On heavily degraded Celeb-Ref images, PFStorer reaches 57.18% ArcFace ID similarity versus 30.01 for the next best (DR2+SPAR), and LMSE 18.37 versus 33.34 for CodeFormer.
- In a user study, PFStorer was voted best 61% of the time versus 25% for the second-best method.
- Adding personalization to the base model raises ID from 33.90 to 57.18 on the heavy split, while the base model alone improves over StableSR on all metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| personalize the restoration model using reference images and adapter blocks with learnable γ (ablation) | Base Model · Celeb-Ref heavy degradation | ID: 33.90 -> 57.18 | Helped |
| add learnable γ to balance personalized and restored features (ablation) | Base Model + ViCo · Celeb-Ref heavy degradation | ID: 53.92 -> 57.18 | Helped |
| replace adapter personalization with DreamBooth fine-tuning (ablation) | Base Model · Celeb-Ref heavy degradation | PSNR: 22.15 -> 21.13 | Hurt |
| replace adapter personalization with DreamBooth fine-tuning (ablation) | Base Model · Celeb-Ref heavy degradation | LMSE: 32.83 -> 37.48 | Hurt |
| replace adapter personalization with ViCo and omit learnable γ (ablation) | Base Model · Celeb-Ref heavy degradation | ID: 57.18 -> 53.92 | Hurt |
| use the improved face-restoration training pipeline instead of StableSR (ablation) | StableSR · Celeb-Ref heavy degradation | ID: 22.87 -> 33.90 | Helped |
| increase generative-regularization weight λGen (ablation) | PFStorer · heavy degradation | perceptual quality | Helped |
| increase generative-regularization weight λGen (ablation) | PFStorer · minor degradation | perceptual quality | Hurt |
Surprising findings
- Standard DreamBooth-style full fine-tuning lowered PSNR below StableSR and raised LMSE, indicating it damages existing restoration priors.
- A larger generative regularization weight deteriorates results when degradation is minor, so the best weight depends on degradation level.
Limitations
- Output is only as faithful as the given reference images; appearance changes between references and input can produce unwanted results.
- Inherits Stable Diffusion's slow sampling and occasional artifacts and hallucinations.
- Evaluation uses a small subset (20 identities, 342 images) of Celeb-Ref due to diffusion compute cost.
Experimental setup
Datasets: FFHQ Celeb-Ref LAION-5B-KNN
Benchmarks: Celeb-Ref
Metrics: PSNR SSIM LPIPS MUSIQ LMSE ID similarity (ArcFace) user study preference
Baselines: CodeFormer DMDNet DR2 StableSR DreamBooth ViCo
Builds on: StableSR DreamBooth ViCo Stable Diffusion
Closest papers by meaning
- FaceMe: Robust Blind Face Restoration with Personal Identification (AAAI 2025 · similarity 0.91)
- Overcoming False Illusions in Real-World Face Restoration with Multi-Modal Guided Diffusion Model (ICLR 2025 · similarity 0.90)
- OSDFace: One-Step Diffusion Model for Face Restoration (CVPR 2025 · similarity 0.90)
- ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration (NeurIPS 2024 · similarity 0.88)
- When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation (CVPR 2024 · similarity 0.87)
- Self-Supervised Selective-Guided Diffusion Model for Old-Photo Face Restoration (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.