Research map / Image generation / Image restoration
Dual Associated Encoder for Face Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DAEFR, a dual-branch codebook-prior face restoration framework that adds an LQ-trained auxiliary encoder, associates HQ and LQ encoder features via a CLIP-style cross-entropy patch-similarity loss, and fuses them with multi-head cross-attention before transformer code prediction.
Problem
Blind face restoration with codebook priors relies on a single HQ-pretrained encoder to encode LQ inputs, which introduces domain bias and imprecise feature representations under severe degradation.
Approach
Add an auxiliary LQ encoder and codebook trained on LQ data, align the HQ and LQ encoder features with a patch-level cross-entropy similarity (association) stage, and fuse them with MHCA to predict HQ code indices.
Key results
- On the real-world BRIAR-Test, DAEFR achieves the lowest FID (90.032) vs. 91.653 for GPEN and 92.072 for VQFR.
- On CelebA-Test, DAEFR attains LPIPS 0.388 and LMD 5.634 (second-best), with PSNR 19.919 below GFP-GAN-class baselines like CodeFormer's 21.449.
- Ablation: two encoders with association and MHCA fusion reduce NIQE from 4.394 (HQ encoder only) to 3.815 and LMD to 4.019.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add an auxiliary LQ encoder alongside the HQ encoder (ablation) | DAEFR · CelebA-Test | NIQE: 4.297 vs 4.394 | Helped |
| Associate the HQ and LQ encoders (ablation) | DAEFR with HQ and LQ encoders · CelebA-Test | NIQE: 4.297 -> 4.290 | Helped |
| Replace linear projection with MHCA feature fusion (ablation) | DAEFR with associated encoders · CelebA-Test | LMD: 4.197 -> 4.019 | Helped |
| Increase LQ feature scale from 0 to 1.0 (ablation) | DAEFR · CelebA-Test | PSNR: 20.040 -> 21.488 | Helped |
| Use cross-entropy rather than L1 association loss (ablation) | DAEFR association · CelebA validation | FID: 70.717 -> 41.230 | Helped |
| Use cross-entropy rather than L2 association loss (ablation) | DAEFR association · CelebA validation | LPIPS: 0.552 -> 0.361 | Helped |
| Use cross-entropy instead of L1 or L2 feature matching (ablation) | DAEFR association · CelebA validation | LPIPS: L1 Loss 0.403; L2 Loss 0.552; Cross-Entropy Loss 0.361 | Mixed |
| Use DAEFR instead of CodeFormer | Face restoration followed by ArcFace verification · LFW face recognition validation, severe degradation | Verification performance (%): 71.67 -> 73.78 | Helped |
Surprising findings
- On CelebA-Test, DAEFR has lower PSNR (19.919) than CodeFormer (21.449), indicating a fidelity trade-off not highlighted in the text.
Limitations
- Fails on extreme face poses because FFHQ has few large-pose samples, leaving codebook features scarce.
- Lower PSNR/SSIM than some baselines on the synthetic CelebA-Test set.
- Multi-stage training (codebook, association, fusion) with many iterations per stage.
Experimental setup
Datasets: FFHQ CelebA-HQ LFW WIDER Face BRIAR
Benchmarks: CelebA-Test LFW-Test WIDER-Test BRIAR-Test LFW face recognition validation
Metrics: FID NIQE LPIPS PSNR SSIM IDA LMD verification accuracy
Baselines: PSFRGAN GFP-GAN GPEN RestoreFormer CodeFormer VQFR DR2
Builds on: VQGAN VQ-VAE CLIP CodeFormer
Closest papers by meaning
- Overcoming False Illusions in Real-World Face Restoration with Multi-Modal Guided Diffusion Model (ICLR 2025 · similarity 0.84)
- DSL-FIQA: Assessing Facial Image Quality via Dual-Set Degradation Learning and Landmark-Guided Transformer (CVPR 2024 · similarity 0.84)
- Controlling Vision-Language Models for Multi-Task Image Restoration (ICLR 2024 · similarity 0.83)
- FaceMe: Robust Blind Face Restoration with Personal Identification (AAAI 2025 · similarity 0.82)
- DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation (NeurIPS 2024 · similarity 0.82)
- ReFIR: Grounding Large Restoration Models with Retrieval Augmentation (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.