Research map / Image generation / Super-resolution
FaithDiff: Unleashing Diffusion Priors for Faithful Image Super-resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FaithDiff, which unfreezes and jointly fine-tunes the VAE encoder, an alignment module, and SDXL diffusion model for faithful latent-diffusion image super-resolution.
Problem
Diffusion-based image super-resolution often produces unfaithful structures because encoder features from degraded inputs mislead a frozen diffusion prior pre-trained on high-quality images.
Approach
Unleash and fine-tune the diffusion model instead of freezing it, add a transformer-based alignment module that fuses penultimate-layer VAE encoder features with the noisy latent, and jointly optimize encoder, alignment module, and diffusion model without ControlNet-style adapters.
Key results
- On RealPhoto60, FaithDiff reports MUSIQ 72.74, CLIPIQA+ 0.5932 and MANIQA 0.6527, the best among compared methods.
- Alignment module ablation on RealPhoto60 improves MUSIQ by 6.07 over removing it (66.67 to 72.74).
- Inference uses 20 steps at 2.55 s per 1024x1024 image versus 7.31-11.44 s for other diffusion SR methods on an A800 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the alignment module (ablation) | FaithDiff · RealPhoto60 | MUSIQ: 66.67 -> 72.74 | Helped |
| pre-train the alignment module (ablation) | FaithDiff · RealPhoto60 | MUSIQ: 69.76 -> 72.74 | Helped |
| use penultimate-layer rather than last-layer LQ features (ablation) | FaithDiff · RealPhoto60 | MUSIQ: 70.05 -> 72.74 | Helped |
| use penultimate-layer rather than last-layer LQ features (ablation) | FaithDiff · DIV2K-Val | LPIPS: 0.3302 -> 0.3080 | Helped |
| fine-tune the encoder and diffusion model jointly instead of fixing the diffusion model (ablation) | FaithDiff · RealPhoto60 | MUSIQ: 69.66 -> 72.74 | Helped |
| fine-tune the encoder and diffusion model jointly instead of fixing the encoder (ablation) | FaithDiff · RealPhoto60 | MUSIQ: 71.11 -> 72.74 | Helped |
| fine-tune the encoder and diffusion model jointly instead of separately (ablation) | FaithDiff · RealPhoto60 | MUSIQ: 69.94 -> 72.74 | Helped |
Surprising findings
- On the severe-degradation synthetic sets, FaithDiff's PSNR and SSIM trail several methods, so the gains are mainly in perceptual metrics rather than fidelity metrics.
Limitations
- PSNR and SSIM on synthetic benchmarks are lower than GAN baselines and some diffusion methods.
- Relies on LLaVA-generated captions for training and text conditioning.
Experimental setup
Datasets: LSDIR DIV2K Flickr2K DIV8K FFHQ DIV2K-Val LSDIR-Val RealPhoto60 RealDeg ICDAR 2024 Occluded RoadText
Benchmarks: DIV2K-Val LSDIR-Val RealPhoto60 RealDeg Occluded RoadText 2024
Metrics: PSNR SSIM LPIPS MUSIQ CLIPIQA+ PaQ-2-PiQ NIQE MANIQA OCR precision OCR recall
Baselines: Real-ESRGAN BSRGAN StableSR DiffBIR PASD SeeSR DreamClear CCSR OSEDiff SUPIR ResShift
Builds on: SDXL PASD SUPIR DiffBIR ControlNet
Closest papers by meaning
- Taming Diffusion Prior for Image Super-Resolution with Domain Shift SDEs (NeurIPS 2024 · similarity 0.91)
- SinSR: Diffusion-Based Image Super-Resolution in a Single Step (CVPR 2024 · similarity 0.90)
- ResDiff: Combining CNN and Diffusion Model for Image Super-resolution (AAAI 2024 · similarity 0.89)
- FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.89)
- One-Step Effective Diffusion Network for Real-World Image Super-Resolution (NeurIPS 2024 · similarity 0.89)
- RAP-SR: RestorAtion Prior Enhancement in Diffusion Models for Realistic Image Super-Resolution (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.