Research map / Image generation / Text-to-image diffusion
Adapting Text-to-Image Generation with Feature Difference Instruction for Generic Image Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffRes, which extracts BLIP-2 Feature Difference Instructions (clean vs. degraded Q-Former feature subtraction) and injects them into a frozen Stable Diffusion V2.1 via a multi-scale FDI adapter for generic image restoration across 10 tasks.
Problem
Diffusion-based T2I models for image restoration suffer from complex training and prompt design, and text/CLIP-derived degradation prompts entangle content and degradation, limiting restoration of complex degradations like haze and mixed degradations.
Approach
Use the difference between BLIP-2 Q-Former features of degraded and clean reference images as a degradation instruction, and inject it through a lightweight multi-scale adapter (channel attention, cross-attention refinement, AdaIN-style visual modulator) into a frozen Stable Diffusion with DDIM inversion, avoiding text prompt design and retraining the T2I model.
Key results
- Single-task setting: achieves 34.55 PSNR on Rain100H deraining vs. 33.91 for DA-CLIP, and best results on most of 9 degradation-specific datasets.
- Multi-task all-in-one on 5 tasks: average PSNR 29.78 vs. 29.55 for InstructIR-5D (best average).
- Mixed-degradation on OutDoor-Rain: 22.30 PSNR / 0.221 LPIPS for composite restoration vs. 21.45 / 0.285 for InstructIR-5D.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiffRes rather than DA-CLIP | single-task restoration · Rain100H | PSNR: 33.91 -> 34.55 | Helped |
| Use DiffRes rather than Xformer | single-task restoration · CBSD68, sigma=50 | PSNR: 29.45 -> 29.17 | Hurt |
| Use DiffRes rather than FocalNet | single-task restoration · DPDD | PSNR: 26.25 -> 26.22 | Hurt |
| Use DiffRes rather than FBCNN | single-task restoration · LIVE1, Q=10 | PSNR: 27.77 -> 27.77 | No clear effect |
| Use DiffRes rather than InstructIR-5D | all-in-one restoration · Average, five restoration tasks | PSNR: 29.55 -> 29.78 | Helped |
| Use composite DiffRes rather than InstructIR-5D | mixed-degradation restoration · RainDS, real-world rainstreak + raindrop | PSNR: 24.26 -> 25.11 | Helped |
| Add a visual modulator using F instead of refined FDI (ablation) | DiffRes · Average of snow, haze, rainstreak, and low-light restoration | PSNR: 15.33 -> 15.90 | Helped |
| Remove channel attention from the visual branch (ablation) | DiffRes · Average of snow, haze, rainstreak, and low-light restoration | PSNR: 15.33 -> 15.28 | Hurt |
| Add noise decoupling (ablation) | DiffRes · Average of snow, haze, rainstreak, and low-light restoration | PSNR: 15.33 -> 21.49 | Helped |
| Keep channel attention in noise-decoupling guidance (ablation) | DiffRes · Average of snow, haze, rainstreak, and low-light restoration | PSNR: 19.32 -> 21.49 | Helped |
| Keep cross-attention in noise decoupling (ablation) | DiffRes · Average of snow, haze, rainstreak, and low-light restoration | PSNR: 20.95 -> 21.49 | Helped |
| Add the visual modulator to noise decoupling (ablation) | DiffRes · Average of snow, haze, rainstreak, and low-light restoration | PSNR: 21.49 -> 24.14 | Helped |
Surprising findings
- Cross-dataset FDI transfer works: FDI from NYU-Rain or OutDoor-Rain (depth-aware rain effects) outperforms synthetic rain-streak datasets on other test sets.
- Cross-task FDI compatibility: FDI from denoising datasets can be used for desnowing.
Limitations
- Degradation classifier and mixed-degradation setting rely on known/predicted degradation types; progressive order shows no clear pattern.
- Cross-dataset generalization is described as only preliminary.
- Per-task performance is not uniformly best (e.g., some tasks such as GoPro deblurring and LOL results are close to or below baselines).
Experimental setup
Datasets: Rain100L Rain100H Rain1400 RESIDE-6K Snow100K RainDrop LOL-v1 MEF DIV2K Flickr2K CBSD68 GoPro
Benchmarks: Rain100H RESIDE Snow100K RainDrop LOL-v1 CBSD68 GoPro DPDD LIVE1 Rain100L SOTS BSD68
Metrics: PSNR SSIM FID LPIPS
Baselines: DA-CLIP PromptIR Restormer MAXIM IR-SDE InstructIR-5D AirNet TransWeather DehazeFormer MPRNet
Builds on: Stable Diffusion BLIP-2 T2I-Adapter DDIM inversion DA-CLIP
Closest papers by meaning
- Improving Image Restoration through Removing Degradations in Textual Representations (CVPR 2024 · similarity 0.90)
- Acquire and then Adapt: Squeezing out Text-to-Image Model for Image Restoration (CVPR 2025 · similarity 0.89)
- Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.89)
- Controlling Vision-Language Models for Multi-Task Image Restoration (ICLR 2024 · similarity 0.88)
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.88)
- Visual-Instructed Degradation Diffusion for All-in-One Image Restoration (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.