Research map / Vision-language and multimodal / Human and embodied modeling
Vision-Language Gradient Descent-driven All-in-One Deep Unfolding Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLU-Net, an all-in-one deep unfolding network whose degradation transform is selected automatically by a CLIP model fine-tuned on degraded image-text pairs, plus a hierarchical feature-level unfolding structure.
Problem
Deep unfolding networks for image restoration require manually chosen degradation matrices per degradation type, so they cannot handle multiple mixed degradations (noise, haze, rain, blur, low light) in a single all-in-one model.
Approach
Replace the manual degradation matrix in the proximal gradient descent step with a VLM-guided attention module: CLIP image features (fine-tuned with adapters on degraded image-text pairs) produce softmax weights that retrieve learnable degradation keys, and a hierarchical multi-level feature unfolding passes information across stages.
Key results
- Under the NHRBL all-in-one setting, VLU-Net averages 30.11 dB PSNR versus 29.55 dB for InstructIR, and reaches 30.84 dB on dehazing (SOTS) versus 27.10 dB for InstructIR.
- On one-by-one Urban100 denoising, VLU-Net improves over InstructIR by about 0.90 dB on average across noise levels.
- Ablation under NHRBL: the hierarchical DUN without CLIP reaches 27.89 dB average, adding tuned CLIP raises it to 28.19 dB, and it is 1.97 dB above DGUNet on the 25.92 dB baseline average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VLU-Net instead of InstructIR | All-in-one restoration · SOTS (NHRBL) | PSNR: 27.10 -> 30.84 | Helped |
| use VLU-Net instead of InstructIR | All-in-one restoration · Rain100L (NHRBL) | PSNR: 36.84 -> 38.54 | Helped |
| use VLU-Net instead of InstructIR | All-in-one restoration · GoPro (NHRBL) | PSNR: 29.40 -> 27.46 | Hurt |
| use VLU-Net instead of InstructIR | All-in-one restoration · LOL (NHRBL) | PSNR: 23.00 -> 22.29 | Hurt |
| use VLU-Net instead of InstructIR | All-in-one restoration · NHR average | PSNR: 32.43 -> 32.70 | Helped |
| remove CLIP guidance (ablation) | Hierarchical VLU-Net · NHRBL ablation average | PSNR: 28.19 -> 27.89 | Hurt |
| use CLIP without degradation fine-tuning (ablation) | Hierarchical VLU-Net · NHRBL ablation average | PSNR: 28.19 -> 27.97 | Hurt |
| fine-tune CLIP on degradation image-text pairs (ablation) | Hierarchical VLU-Net with CLIP · NHRBL ablation average | PSNR: 27.97 -> 28.19 | Helped |
| use hierarchical feature-level unfolding instead of the previous DUN (ablation) | Image restoration DUN · NHRBL ablation average | PSNR: 25.92 -> 27.89 | Helped |
| use VLU-Net instead of PromptIR | All-in-one restoration · Random noise average | PSNR: 30.51 -> 30.75 | Helped |
Surprising findings
- The hierarchical DUN without CLIP guidance already achieves results comparable to state-of-the-art all-in-one methods, indicating the feature-level unfolding contributes most of the gain beyond the VLM.
Limitations
- Degradation types are limited to a fixed set of seen degradations (noise, haze, rain, blur, low light); generalization to unseen degradations is only partly tested via random noise levels.
- Some results lag behind baselines, e.g., on low-light LOL and GoPro deblurring VLU-Net is below PromptIR/InstructIR/MambaIR.
Experimental setup
Datasets: BSD400 WED RESIDE-β-OTS Rain100L GoPro LOL CBSD68 Urban100 SOTS-Outdoor
Benchmarks: SOTS-Outdoor Rain100L CBSD68 GoPro LOL Urban100
Metrics: PSNR SSIM
Baselines: DGUNet Restormer MambaIR AirNet TransWeather IDR PromptIR Gridformer InstructIR NDR
Builds on: Deep Unfolding Networks DGUNet CLIP Restormer PromptIR
Closest papers by meaning
- Controlling Vision-Language Models for Multi-Task Image Restoration (ICLR 2024 · similarity 0.89)
- DDR: Exploiting Deep Degradation Response as Flexible Image Descriptor (NeurIPS 2024 · similarity 0.87)
- GenDeg: Diffusion-based Degradation Synthesis for Generalizable All-In-One Image Restoration (CVPR 2025 · similarity 0.86)
- UARE: A Unified Vision-Language Model for Image Quality Assessment, Restoration, and Enhancement (CVPR 2026 · similarity 0.85)
- Omni-Kernel Network for Image Restoration (AAAI 2024 · similarity 0.85)
- Degradation-Aware Feature Perturbation for All-in-One Image Restoration (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.