Research map / Image generation / Image restoration
Visual-Instructed Degradation Diffusion for All-in-One Image Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defusion is an all-in-one image restoration framework that guides a diffusion model operating in the degradation space with visual instructions built by applying degradations to standard visual grounds (TE42 charts).
Problem
All-in-one image restoration across mixed or unknown degradations, where implicit priors lack generalizability and text instructions are ambiguous and weakly aligned with low-level visual degradation patterns.
Approach
Visual instructions are made by applying the degradation process to standardized visual grounds, tokenized with a quantized autoencoder, and used to guide a diffusion model that denoises the LQ-to-HQ difference (degradation space) rather than the image space, conditioned on LQ images via an Image Restoration Bridge and a Visual Instruction Adapter.
Key results
- Defusion reaches 34.53 PSNR on GoPro motion deblurring, above the best listed baseline (Stripformer/DiffIR at ~33.2).
- On NH-HAZE dehazing, Defusion achieves 22.87 PSNR / 0.853 SSIM versus 22.57 / 0.795 for the all-in-one Restormer.
- Ablation on the average of deraining, dehazing and desnowing drops from 33.83 PSNR to 31.72 when replacing degradation diffusion with naive image-space diffusion, and to 30.86 without instruction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace degradation diffusion with naive diffusion (ablation) | Defusion · Average of deraining, dehazing and desnowing | PSNR: 31.72 -> 33.83 | Helped |
| replace visual grounding with blank grounding (ablation) | Defusion · Average of deraining, dehazing and desnowing | PSNR: 30.89 -> 33.83 | Helped |
| replace visual grounding with simple striped grounding (ablation) | Defusion · Average of deraining, dehazing and desnowing | PSNR: 31.64 -> 33.83 | Helped |
| replace visual instructions with text instructions (ablation) | Defusion · Average of deraining, dehazing and desnowing | PSNR: 32.57 -> 33.83 | Helped |
| add visual instructions (ablation) | Defusion · Average of deraining, dehazing and desnowing | PSNR: 30.86 -> 33.83 | Helped |
| use Defusion instead of DiffIR | all-in-one image restoration · GoPro | PSNR: 33.20 -> 34.53 | Helped |
| use Defusion instead of NAFNet* | all-in-one image restoration · DPDD | PSNR: 28.21 -> 29.68 | Helped |
| use Defusion instead of WeatherDiff | all-in-one image restoration · Snow100K-L | PSNR: 30.43 -> 32.11 | Helped |
| use Defusion instead of DAGN | image restoration · LIVE1 (q=10) | PSNR: 27.95 -> 27.77 | Hurt |
| use Defusion instead of Restormer* | all-in-one image restoration · SIDD | PSNR: 40.60 -> 40.89 | Helped |
| use Defusion instead of Restormer* | all-in-one image restoration · RealBlur-J | PSNR: 30.91 -> 32.39 | Helped |
Surprising findings
- Injecting LQ features into the U-Net decoder via ControlNet-style adapters degraded performance, which the authors attribute to diverging features in the encoder.
Limitations
- Visual instruction tokenizer and grounding pool rely on hand-chosen TE42 chart constructs; generality to unseen degradations is asserted rather than extensively tested.
- Text-based instructions are not used for training in the main results; authors note text can still be injected through the pretrained cross-attention.
Experimental setup
Datasets: Rain1400 Outdoor-Rain LHP RESIDE Dense-Haze Snow100K RealSnow RainDrop RainDS SIDD GoPro RealBlur
Benchmarks: GoPro DPDD Snow100K-L Dense-Haze Rain1400 RainDrop SIDD LIVE1 NH-HAZE LHP RealSnow RealBlur-J
Metrics: PSNR SSIM LPIPS
Baselines: Restormer NAFNet PromptIR AirNet DA-CLIP MPerceiver TransWeather WeatherDiff MPRNet DiffIR
Builds on: latent diffusion / Stable Diffusion IP-Adapter VQ-VAE DDIM InstructPix2Pix
Closest papers by meaning
- UP-Restorer: When Unrolling Meets Prompts for Unified Image Restoration (AAAI 2025 · similarity 0.89)
- Residual Diffusion Deblurring Model for Single Image Defocus Deblurring (AAAI 2025 · similarity 0.88)
- Continuous Expert Assembly: Instance-Conditioned Low-Rank Residuals for All-in-One Image Restoration (NeurIPS 2026 · similarity 0.88)
- Learning Where and What to Restore for Composite Image Restoration (NeurIPS 2026 · similarity 0.88)
- GenDeg: Diffusion-based Degradation Synthesis for Generalizable All-In-One Image Restoration (CVPR 2025 · similarity 0.88)
- Test-Time Degradation Adaptation for Open-Set Image Restoration (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.