Research map / Image generation / Image restoration
ReFIR: Grounding Large Restoration Models with Retrieval Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReFIR, a training-free retrieval-augmented framework that injects nearest-neighbor retrieved high-quality reference images into diffusion-based large restoration models (SeeSR, SUPIR) to reduce hallucination.
Problem
Large restoration models (LRMs) built on diffusion priors hallucinate incorrect textures when images are severely degraded, because they rely only on limited internal knowledge in their weights.
Approach
Retrieve content-relevant HQ images via cosine nearest-neighbor lookup in a feature space and inject their decoder self-attention features into the LRM's denoising chain through separate intra/inter-chain attention, spatial adaptive gating, and AdaIN distribution alignment, with no training.
Key results
- On CUFED5, SUPIR+ReFIR improves FID from 168.26 to 148.69 (19.57 drop) and SeeSR+ReFIR gains 0.38 dB PSNR over SeeSR (20.32 vs 19.94).
- On RealPhoto60, SeeSR+ReFIR improves NIQE by 0.2866 and MUSIQ by 1.59 over SeeSR.
- ReFIR adds about 1.38x GPU memory and roughly doubles inference time for SUPIR (146.4s to 322.8s on 2048x2048 input on A100).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ReFIR | SeeSR · CUFED5 | PSNR: +0.38 | Helped |
| add ReFIR | SUPIR · CUFED5 | FID: +19.57 | Helped |
| add ReFIR | SeeSR · RealPhoto60 | MUSIQ: +1.59 | Helped |
| replace spatial adaptive gating with fixed scalar weights (ablation) | SUPIR+ReFIR · CUFED5 | NIQE: 0.18 NIQE drop | Hurt |
| remove distribution alignment (ablation) | SUPIR+ReFIR · CUFED5 | FID: 4.36 FID drop | Hurt |
| perform cross-image injection only on the encoder (ablation) | SUPIR+ReFIR · CUFED5 | FID: 19.57 FID drops | Hurt |
| use multiple reference images instead of one (ablation) | SUPIR+ReFIR · CUFED5 | FID: 2.08 improvement in FID | Helped |
| use the ground-truth HQ image as the reference (ablation) | SUPIR+ReFIR · CUFED5 | restoration performance | Helped |
| use the LQ image itself as its reference rather than no retrieval (ablation) | SUPIR+ReFIR · CUFED5 | restoration performance | Helped |
| use a randomly selected HQ reference (ablation) | SUPIR+ReFIR · CUFED5 | restoration performance | Hurt |
| use too small a control scale s (ablation) | SUPIR+ReFIR · CUFED5 | restoration performance | Hurt |
| use too large a control scale s (ablation) | SUPIR+ReFIR · CUFED5 | restoration performance | Hurt |
Surprising findings
- Using the LQ image itself as its own reference gives a slight improvement over no retrieval, attributed to distribution alignment regularization.
- Randomly chosen high-quality references cause a large performance drop, suggesting content correlation matters more than image quality.
Limitations
- Increased GPU memory and inference latency due to the second denoising chain and extra reference image.
- Simple semantic-feature retriever is not texture-aware; better retrievers are left to future work.
- Applied to slightly degraded images unnecessarily; no adaptive decision on when to use retrieval.
Experimental setup
Datasets: CUFED5 WR-SR DIV2K RealPhoto60
Benchmarks: CUFED5 WR-SR RealPhoto60
Metrics: PSNR SSIM LPIPS NIQE FID MUSIQ CLIPIQA
Baselines: C2-Matching DATSR MRefSR BSRGAN Real-ESRGAN StableSR DiffBIR PASD CCSR SeeSR SUPIR
Builds on: SUPIR SeeSR ControlNet AdaIN Real-ESRGAN
Closest papers by meaning
- LearnIR: Learnable Posterior Sampling for Real-World Image Restoration (ICLR 2026 · similarity 0.90)
- DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation (NeurIPS 2024 · similarity 0.89)
- Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (CVPR 2024 · similarity 0.88)
- RAP-SR: RestorAtion Prior Enhancement in Diffusion Models for Realistic Image Super-Resolution (AAAI 2025 · similarity 0.87)
- Learning Diffusion Texture Priors for Image Restoration (CVPR 2024 · similarity 0.87)
- From Zero to Detail: Deconstructing Ultra-High-Definition Image Restoration from Progressive Spectral Perspective (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.