Research map / Image generation / Editing and style transfer
Overcoming False Illusions in Real-World Face Restoration with Multi-Modal Guided Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MGFR, a multi-modal guided diffusion face restoration method that combines attribute text prompts, a high-quality reference image and ArcFace identity embedding via a dual-control adapter trained in two stages, and releases the Reface-HQ dataset.
Problem
Generative real-world face restoration from severely degraded inputs hallucinates facial attributes and identities that differ from the true person.
Approach
Augment a Stable Diffusion-based restoration model with a dual-control adapter (LQ control adapter plus reference control adapter) that jointly uses attribute text prompts, an HQ reference image and ArcFace identity features, trained in two stages (text-guided first, then reference/identity), with negative-quality and negative-attribute prompts applied via classifier-free guidance.
Key results
- On CelebA-Test Real-SR x4, the w/o-reference model reaches ManIQA 0.6854, ClipIQA 0.8244 and MUSIQ 76.22, the best among compared methods on these non-reference metrics.
- On Reface-Test x16, the reference-guided model achieves ArcFace ID distance 0.7401 vs 0.8658 for CodeFormer, a lower identity distance than baselines at the x16 setting (note: a lower x8 ID of 0.5166 is also reported).
- Ablation shows adding positive attribute prompts raises MUSIQ from 49.97 to 73.26 and ClipIQA from 0.2568 to 0.7008 on CelebA-Test.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Link-UL from the UNet to LCA (ablation) | MGFR dual-control adapter · Real-SR (×8) | ManIQA: 0.5931 -> 0.6535 | Helped |
| add upward LCA-to-RCA information flow (Link-LR) (ablation) | MGFR dual-control adapter · Real-SR (×8) | ClipIQA: 0.6645 -> 0.8147 | Helped |
| add negative-quality prompt alongside the positive attribute prompt (ablation) | MGFR · Real-SR (×8) | ManIQA: 0.6441 -> 0.6656 | Helped |
| add negative-attribute prompt alongside the positive attribute prompt (ablation) | MGFR · Real-SR (×8) | ManIQA: 0.6441 -> 0.6437 | No clear effect |
| add negative-attribute prompt when positive and negative-quality prompts are already used (ablation) | MGFR · Real-SR (×8) | ManIQA: 0.6656 -> 0.6776 | Helped |
| use 20K rather than 10K training samples (ablation) | MGFR · Real-SR (×8) | ManIQA: 0.6388 -> 0.6535 | Helped |
| use reference-image and identity guidance rather than the text-guided model without reference | MGFR · Reface-Test Real-SR (×8) | ManIQA | Helped |
| use reference-image and identity guidance rather than the text-guided model without reference | MGFR · Reface-Test | perceived restoration quality | Helped |
| use negative-quality samples and prompts in training and inference | MGFR · face restoration | perceived image quality | Helped |
| remove ArcFace identity embedding (ablation) | MGFR · face restoration | identity fidelity | Hurt |
| use attribute prompts that strongly contradict the low-resolution input | MGFR · attribute-guided face restoration | restoration quality | No clear effect |
Surprising findings
- Full-reference metrics such as PSNR and SSIM do not track perceptual quality; the authors note MGFR scores lower PSNR/SSIM than several baselines while leading non-reference metrics.
- Adding both negative-quality and negative-attribute prompts together reduces PSNR and SSIM relative to positive-only prompts in the ablation.
Limitations
- Attribute text control is limited to fixed template prompts and unseen attribute labels are ineffective.
- Only a single reference image is supported; multiple low-quality references are left to future work.
- Inference with classifier-free guidance is slower and heavier than several diffusion baselines.
Experimental setup
Datasets: FFHQ Reface-HQ CelebA-HQ test (CelebA-Test) WebPhoto-Test
Benchmarks: CelebA-Test Reface-Test WebPhoto-Test
Metrics: LPIPS PSNR SSIM ManIQA ClipIQA MUSIQ ArcFace ID distance user study
Baselines: PSFRGAN GPEN VQFR CodeFormer DR2 DiffBIR BFRffusion ASFFNet DMDNet SUPIR
Builds on: Stable Diffusion ControlNet DiffBIR classifier-free guidance
Closest papers by meaning
- PFStorer: Personalized Face Restoration and Super-Resolution (CVPR 2024 · similarity 0.90)
- ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration (NeurIPS 2024 · similarity 0.89)
- OSDFace: One-Step Diffusion Model for Face Restoration (CVPR 2025 · similarity 0.88)
- Self-Supervised Selective-Guided Diffusion Model for Old-Photo Face Restoration (NeurIPS 2025 · similarity 0.88)
- FaceMe: Robust Blind Face Restoration with Personal Identification (AAAI 2025 · similarity 0.88)
- WaveFace: Authentic Face Restoration with Efficient Frequency Recovery (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.