Research map / Image generation / Image restoration
$InterLCM$: Low-Quality Images as Intermediate States of Latent Consistency Models for Effective Blind Face Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Treats low-quality face images as an intermediate state of a 4-step latent consistency model (starting at step 2), adds a CLIP-based Visual Encoder and a ControlNet-style Spatial Encoder, and trains with L1, perceptual and adversarial losses for blind face restoration.
Problem
Diffusion-prior blind face restoration suffers from weak semantic consistency (identity, structure, color) and slow multi-step sampling, which also prevents effective use of perceptual losses on the final output.
Approach
Insert the LQ image into the LCM trajectory at an intermediate step (after the first noise addition, starting from step 2) so the consistency model maps it directly to an HQ image in few steps, conditioned on visual CLIP-derived embeddings and spatial features, enabling image-domain perceptual and adversarial losses end-to-end.
Key results
- On synthetic CelebA-Test, the method achieves best LPIPS (0.223), MUSIQ (76.58), IDS (33.64) and SSIM (0.718) among compared methods.
- On real-world WebPhoto-Test and WIDER-Test, it reports best FID (75.48 and 35.43) and MUSIQ (75.88 and 76.29) among the compared methods.
- Inference takes 0.421 s per 512x512 image on an RTX 3090, faster than diffusion baselines PGDiff (14.768 s) and WaveFace (19.370 s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InterLCM with LCM rather than the listed baseline methods | Blind face restoration · CelebA-Test | MUSIQ: 75.55 -> 76.58 | Helped |
| Use InterLCM with LCM rather than the listed baseline methods | Blind face restoration · WebPhoto-Test | FID: 75.77 -> 75.48 | Helped |
| Use InterLCM with LCM rather than the listed baseline methods | Blind face restoration · WIDER-Test | MUSIQ: 74.01 -> 76.29 | Helped |
| Add the Spatial Encoder (ablation) | InterLCM with Visual Encoder, 2nd-step start · LFW-Test | FID: 69.99 -> 51.32 | Helped |
| Use a Visual Encoder rather than NullText conditioning (ablation) | InterLCM with Spatial Encoder, 2nd-step start · LFW-Test | FID: 55.56 -> 51.32 | Helped |
| Use a Visual Encoder rather than Text conditioning (ablation) | InterLCM with Spatial Encoder, 2nd-step start · LFW-Test | FID: 55.07 -> 51.32 | Helped |
| Start at the 1st step rather than the 2nd step (ablation) | InterLCM with Visual Encoder and Spatial Encoder · LFW-Test | MUSIQ: 71.50 -> 76.16 | Hurt |
| Start at the 3rd step rather than the 2nd step (ablation) | InterLCM with Visual Encoder and Spatial Encoder · LFW-Test | FID: 50.48 -> 51.32 | No clear effect |
| Start at the 4th step rather than the 2nd step (ablation) | InterLCM with Visual Encoder and Spatial Encoder · WIDER-Test | FID: 35.43 -> 50.70 | Hurt |
| Add perceptual loss (ablation) | InterLCM trained with L1 loss · LFW-Test | FID: 87.12 -> 57.57 | Helped |
| Add adversarial loss (ablation) | InterLCM trained with L1 and perceptual losses · LFW-Test | FID: 57.57 -> 51.32 | Helped |
| Use 4-step LCM rather than 2-step LCM (ablation) | InterLCM · CelebA-Test | FID: 49.19 -> 45.38 | Helped |
Surprising findings
- Naive ControlNet achieves better FID on LFW-Test (35.43) than the proposed Spatial Encoder, though with lower MUSIQ, indicating a fidelity-quality trade-off.
- A 2-step LCM backbone performs slightly better than the 4-step LCM on some metrics (e.g., LFW MUSIQ 76.24 vs 76.16).
Limitations
- Fails to generate realistic hands in images containing hands, attributed to limited such images in FFHQ.
- Struggles with complex tattoo text and fine intricate textures due to scarcity in training data.
- Less effective on general natural image restoration with complex textures.
Experimental setup
Datasets: FFHQ CelebA-Test LFW-Test WebPhoto-Test WIDER-Test
Benchmarks: CelebA-Test LFW-Test WebPhoto-Test WIDER-Test
Metrics: LPIPS FID MUSIQ IDS PSNR SSIM inference time
Baselines: PULSE DFDNet PSFRGAN GFPGAN GPEN RestoreFormer VQFR CodeFormer DR2 DifFace PGDiff WaveFace
Builds on: latent consistency model ControlNet CodeFormer Stable Diffusion LCM-LoRA
Closest papers by meaning
- ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration (NeurIPS 2024 · similarity 0.90)
- Learning Degradation-unaware Representation with Prior-based Latent Transformations for Blind Face Restoration (CVPR 2024 · similarity 0.87)
- Consistency Purification: Effective and Efficient Diffusion Purification towards Certified Robustness (NeurIPS 2024 · similarity 0.87)
- FaceMe: Robust Blind Face Restoration with Personal Identification (AAAI 2025 · similarity 0.87)
- Blind Face Restoration under Extreme Conditions: Leveraging 3D-2D Prior Fusion for Superior Structural and Texture Recovery (AAAI 2024 · similarity 0.87)
- A Modular Conditional Diffusion Framework for Image Reconstruction (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.