Research map / 3D vision / Face and shape
Blind Face Restoration under Extreme Conditions: Leveraging 3D-2D Prior Fusion for Superior Structural and Texture Recovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FREx, which fuses implicit 3D geometry priors from EG3D with 2D texture priors from StyleGAN2 via a pose-conditioned Adaptive Weighting Block and cross-attention to restore severely degraded faces under extreme poses.
Problem
Blind face restoration under extreme degradations and extreme head poses, where 2D generative/reference priors produce erroneous facial structures.
Approach
Combine structure-accurate 3D priors (pretrained EG3D decoder, with a 2D-encoder-guided multi-scale adversarial refinement) with texture-rich StyleGAN2 2D priors, fused by an Adaptive Weighting Block whose modulation is conditioned on estimated head pose, plus cross-attention between 3D and 2D features.
Key results
- On Balanced CelebAHQ Test, FREx reaches LPIPS 0.4671, FID 39.58, PSNR 20.95, SSIM 0.5890, outperforming CodeFormer (FID 44.83) and GLEAN (FID 75.37).
- Ablations show removing the adaptive weight block raises FID from 39.58 to 48.35, and removing 2D-3D cross-attention raises it to 48.34.
- Removing the 3D geometry prior lowers PSNR from 20.95 to 20.63 and raises FID to 41.01.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FREx with fused 3D geometry and 2D texture priors | FREx · Balanced CelebAHQ Test | LPIPS: 0.4981 -> 0.4671 | Helped |
| Add the 3D geometry prior (ablation) | FREx · Balanced CelebAHQ Test | LPIPS: 0.4703 -> 0.4671 | Helped |
| Add the adaptive weighting block (ablation) | FREx · Balanced CelebAHQ Test | FID: 48.35 -> 39.58 | Helped |
| Add 2D-3D cross-attention (ablation) | FREx · Balanced CelebAHQ Test | FID: 48.34 -> 39.58 | Helped |
| Add the multi-scale discriminator (ablation) | FREx · Balanced CelebAHQ Test | FID: 45.56 -> 39.58 | Helped |
| Add facial structure loss (ablation) | FREx · Balanced CelebAHQ Test | FID: 49.42 -> 39.58 | Helped |
Surprising findings
- The 3D encoder alone restores accurate structures but yields oversmoothed results lacking fine detail, showing 2D priors are still needed for texture.
Limitations
- Produces blurry results on the top of the head due to face misalignment in the pretrained EG3D decoder.
- The EG3D decoder can only generate a specific range of facial features; retraining it is left for future work.
Experimental setup
Datasets: FFHQ CelebAHQ CelebA WIDERFACE
Benchmarks: Balanced CelebAHQ Test real-world set from CelebA and WIDERFACE (300 images)
Metrics: LPIPS FID NIQE Deg. (ArcFace identity similarity) PSNR SSIM
Baselines: GLEAN GPEN GFP-GAN RestoreFormer CodeFormer VQFR Bilinear
Builds on: EG3D StyleGAN2 GLEAN pSp-style W+ inversion ArcFace
Closest papers by meaning
- FaceMe: Robust Blind Face Restoration with Personal Identification (AAAI 2025 · similarity 0.87)
- Learning Degradation-unaware Representation with Prior-based Latent Transformations for Blind Face Restoration (CVPR 2024 · similarity 0.87)
- FreeUV: Ground-Truth-Free Realistic Facial UV Texture Recovery via Cross-Assembly Inference Strategy (CVPR 2025 · similarity 0.87)
- $InterLCM$: Low-Quality Images as Intermediate States of Latent Consistency Models for Effective Blind Face Restoration (ICLR 2025 · similarity 0.87)
- Discrete Prior-Based Temporal-Coherent Content Prediction for Blind Face Video Restoration (AAAI 2025 · similarity 0.86)
- Overcoming False Illusions in Real-World Face Restoration with Multi-Modal Guided Diffusion Model (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.