Research map / Image generation / Editing and style transfer
ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReF-LDM, a latent diffusion model for face restoration that uses multiple unaligned HQ reference images via a CacheKV mechanism and a timestep-scaled identity loss, and releases the FFHQ-Ref dataset.
Problem
Blind face image restoration often produces HQ faces that do not match the true identity when the LQ input is severely degraded; existing reference-based methods rely on landmark detection that fails on degraded inputs.
Approach
Cache the keys and values of multiple unaligned reference images from one U-net pass and concatenate them into every self-attention layer of the denoising U-net, combined with an identity loss scaled by sqrt(alpha_bar_t) so that identity supervision is down-weighted at noisy timesteps.
Key results
- On FFHQ-Ref-Severe, ReF-LDM reaches IDS 0.676 versus 0.323 for CodeFormer and 0.185 for DMDNet.
- On FFHQ-Ref-Moderate, ReF-LDM achieves IDS 0.840 and FID 33.05, while DMDNet gets IDS 0.810.
- CacheKV reaches IDS 0.65 with 12.15 s inference time versus 58.36 s for spatial-concatenation (IDS 0.69), i.e. about 20% of the time and 39% of the memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ReF-LDM with reference images instead of CodeFormer | FFHQ-Ref-Severe restoration · FFHQ-Ref-Severe | IDS: 0.323 -> 0.676 | Helped |
| use ReF-LDM with reference images instead of DMDNet | FFHQ-Ref-Moderate restoration · FFHQ-Ref-Moderate | IDS: 0.810 -> 0.840 | Helped |
| use ReF-LDM with reference images instead of DMDNet | CelebA-Test-Ref restoration · CelebA-Test-Ref | IDS: 0.752 -> 0.779 | Helped |
| replace channel-concatenation with CacheKV (ablation) | ReF-LDM · FFHQ-Ref-Severe | IDS: 0.23 -> 0.65 | Helped |
| replace cross-attention with CacheKV (ablation) | ReF-LDM · FFHQ-Ref-Severe | IDS: 0.23 -> 0.65 | Helped |
| replace CacheKV with spatial-concatenation (ablation) | ReF-LDM · FFHQ-Ref-Severe | IDS: 0.65 -> 0.69 | Helped |
| replace spatial-concatenation with CacheKV (ablation) | ReF-LDM · FFHQ-Ref-Severe | inference time: 58.36 -> 12.15 | Helped |
| increase reference images from 1 to 3 (ablation) | ReF-LDM · FFHQ-Ref-Severe | IDS: 0.52 -> 0.62 | Helped |
| increase reference images from 3 to 5 (ablation) | ReF-LDM · FFHQ-Ref-Severe | IDS: 0.62 -> 0.65 | Helped |
| increase reference images from 5 to 8 (ablation) | ReF-LDM · FFHQ-Ref-Severe | IDS: 0.65 -> 0.66 | Helped |
| increase reference images from 5 to 8 (ablation) | ReF-LDM · FFHQ-Ref-Severe | inference time: 12.03 -> out-of-memory | Hurt |
Surprising findings
- Naively adding the ArcFace identity loss significantly worsens image quality, motivating the timestep scaling.
- Channel-concatenation and cross-attention fail to use reference images to improve identity similarity.
Limitations
- Artifacts when the face is occluded.
- Unnatural eyes for side-face poses, and poor results when references lack similar poses.
- LPIPS slightly worse than baselines due to background differences, and illumination of references can bias output colour.
Experimental setup
Datasets: FFHQ FFHQ-Ref CelebA-Test CelebA-HQ CelebRef-HQ
Benchmarks: FFHQ-Ref-Severe FFHQ-Ref-Moderate CelebA-Test-Ref
Metrics: IDS fLPIPS LPIPS FID NIQE
Baselines: CodeFormer VQFR DAEFR LDM DMDNet
Builds on: latent diffusion model VQGAN ArcFace classifier-free guidance MasaCtrl
Closest papers by meaning
- FaceMe: Robust Blind Face Restoration with Personal Identification (AAAI 2025 · similarity 0.91)
- $InterLCM$: Low-Quality Images as Intermediate States of Latent Consistency Models for Effective Blind Face Restoration (ICLR 2025 · similarity 0.90)
- Overcoming False Illusions in Real-World Face Restoration with Multi-Modal Guided Diffusion Model (ICLR 2025 · similarity 0.89)
- OSDFace: One-Step Diffusion Model for Face Restoration (CVPR 2025 · similarity 0.88)
- PFStorer: Personalized Face Restoration and Super-Resolution (CVPR 2024 · similarity 0.88)
- Self-Supervised Selective-Guided Diffusion Model for Old-Photo Face Restoration (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.