Research map / Image generation / Editing and style transfer
FaceMe: Robust Blind Face Restoration with Personal Identification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FaceMe, a fine-tuning-free personalized blind face restoration method that uses an identity encoder (CLIP + ArcFace features) to feed identity embeddings into an SDXL-based diffusion model with ControlNet, trained on FFHQ with synthesized multi-pose reference images (FFHQRef).
Problem
Blind face restoration often fails to preserve the true identity of the person, and existing personalized restoration methods require per-identity fine-tuning on 5-20 reference images.
Approach
Encode each reference face with CLIP and ArcFace features into identity embeddings that replace the "face" token embedding in a diffusion prompt; concatenate embeddings to support any number of references; train in two stages (identity encoder first, then ControlNet with frozen encoder); and build a pose/expression-diverse reference pool by synthesizing references with Arc2Face.
Key results
- On CelebRef-HQ, FaceMe achieves PSNR 24.37, FID 51.01, LMD 2.908 and IDS 0.647, the best among compared methods on those metrics.
- On WIDER-Test, FaceMe reaches the best FID of 33.59 and on LFW-Test a FID of 43.95 (best).
- Two-stage training improves IDS from 0.534 (one-stage) to 0.647 and reduces FID from 61.84 to 51.01.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FaceMe instead of the strongest listed baseline | blind face restoration · CelebRef-HQ | IDS: 0.615 -> 0.647 | Helped |
| use FaceMe instead of the strongest listed baseline | blind face restoration · LFW-Test | FID: 46.99 -> 43.95 | Helped |
| use FaceMe instead of the strongest listed baseline | blind face restoration · WIDER-Test | FID: 38.47 -> 33.59 | Helped |
| use FaceMe instead of the strongest listed baseline | blind face restoration · WebPhoto-Test | FID: 77.44 -> 92.46 | Hurt |
| increase reference images from one to four (ablation) | FaceMe · CelebRef-HQ | IDS: 0.620 -> 0.634 | Helped |
| increase reference images from one to four (ablation) | FaceMe · CelebRef-HQ | PSNR: 24.65 -> 24.52 | Hurt |
| include ground truth as a reference (ablation) | FaceMe · CelebRef-HQ | IDS: 0.620 -> 0.641 | Helped |
| replace two-stage training with one-stage joint training (ablation) | FaceMe · CelebRef-HQ | IDS: 0.647 -> 0.534 | Hurt |
| replace two-stage training with one-stage joint training (ablation) | FaceMe · CelebRef-HQ | PSNR: 24.37 -> 22.23 | Hurt |
| add the identity encoder (ablation) | FaceMe · CelebRef-HQ | IDS: 0.568 -> 0.647 | Helped |
| add the identity encoder (ablation) | FaceMe · CelebRef-HQ | FID: 59.87 -> 51.01 | Helped |
Surprising findings
- Removing the identity encoder gives slightly higher PSNR (24.88 vs 24.37) despite worse identity and perceptual metrics.
Limitations
- Real-world test references are synthesized from CodeFormer-restored inputs via Arc2Face, so they are low quality and posed, not true user photos.
- PGDiff could not be evaluated on real-world datasets due to long inference time.
Experimental setup
Datasets: FFHQ FFHQRef CelebRef-HQ LFW-Test WebPhoto-Test WIDER-Test
Benchmarks: CelebRef-HQ LFW-Test WebPhoto-Test WIDER-Test
Metrics: PSNR SSIM LPIPS MUSIQ FID LMD IDS
Baselines: GFP-GAN CodeFormer GPEN VQFR DifFace DR2 PGDiff DMDNet
Builds on: PhotoMaker ControlNet SDXL Arc2Face DifFace
Closest papers by meaning
- PFStorer: Personalized Face Restoration and Super-Resolution (CVPR 2024 · similarity 0.91)
- ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration (NeurIPS 2024 · similarity 0.91)
- OSDFace: One-Step Diffusion Model for Face Restoration (CVPR 2025 · similarity 0.90)
- WaveFace: Authentic Face Restoration with Efficient Frequency Recovery (CVPR 2024 · similarity 0.89)
- Overcoming False Illusions in Real-World Face Restoration with Multi-Modal Guided Diffusion Model (ICLR 2025 · similarity 0.88)
- Blind Face Restoration under Extreme Conditions: Leveraging 3D-2D Prior Fusion for Superior Structural and Texture Recovery (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.