Research map / Vision-language and multimodal / CLIP and zero-shot

Controlling Vision-Language Models for Multi-Task Image Restoration

ICLR 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: Fine-grained improvements to CLIP

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DA-CLIP, a degradation-aware fine-tuned CLIP that predicts degradation and clean-content embeddings from low-quality images to steer diffusion (IR-SDE) and MSE-based restoration models for unified and degradation-specific image restoration.

Problem

Pretrained vision-language models like CLIP degrade on low-level inputs, and existing all-in-one image restoration methods lack discriminative degradation context and semantic guidance for restoring images with diverse corruptions.

Approach

Fine-tune CLIP's image encoder with a degradation classification/contrastive objective against frozen text encoder embeddings of degradation labels and clean captions, then inject the predicted degradation and HQ content embeddings into restoration networks via prompt modules and cross-attention.

Key results

• On the unseen light rain dataset Rain100L, DA-CLIP reaches 36.61 PSNR / 0.025 LPIPS / 11.98 FID versus 32.77 PSNR / 0.081 LPIPS / 43.31 FID for PromptIR.
• Adding DA-CLIP embeddings to NAFNet raises unified-restoration PSNR from 26.34 to 27.22 and lowers FID from 55.68 to 47.94.
• On GoPro, MAXIM's PSNR is about 2 dB higher than DA-CLIP's, but DA-CLIP achieves better perceptual scores (LPIPS 0.058 vs 0.089; FID 6.15 vs 11.57).

Measured effects reported in the paper

Surprising findings

• Ground-truth text embeddings from the frozen text encoder perform similarly to predicted image embeddings from DA-CLIP.
• MSE-based models such as PromptIR and NAFNet achieve higher PSNR/SSIM than the diffusion-based DA-CLIP model on several tasks.

Limitations

• Cannot restore images with multiple concurrent degradations because training pairs carry only one degradation label.
• Poor generalization to real-world images despite gains on Rain100L.
• Cannot describe degradation levels in text.

Experimental setup

Datasets: Rain100H Rain100L GoPro mixed degradation dataset (ten degradation types)

Benchmarks: Rain100L GoPro mixed degradation dataset (ten degradation types)

Metrics: PSNR SSIM LPIPS FID

Baselines: AirNet PromptIR IR-SDE NAFNet MAXIM Restormer CLIP + finetune

Builds on: CLIP IR-SDE NAFNet PromptIR AirNet

Closest papers by meaning

• Vision-Language Gradient Descent-driven All-in-One Deep Unfolding Networks (CVPR 2025 · similarity 0.89)
• Adapting Text-to-Image Generation with Feature Difference Instruction for Generic Image Restoration (CVPR 2025 · similarity 0.88)
• Mint: A Simple Test-Time Adaptation of Vision-Language Models against Common Corruptions (NeurIPS 2025 · similarity 0.88)
• Boosting Image Restoration via Priors from Pre-trained Models (CVPR 2024 · similarity 0.88)
• Improving Image Restoration through Removing Degradations in Textual Representations (CVPR 2024 · similarity 0.87)
• Learning from History: Task-agnostic Model Contrastive Learning for Image Restoration (AAAI 2024 · similarity 0.87)