Research map / Image generation / Fast diffusion sampling

Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks

CVPR 2024 · Image generation › Fast diffusion sampling · approach: Diffusion models as perception tools · 29 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Diff-Plugin adds lightweight task-specific plugins (a dual-branch Task-Plugin) to a frozen pre-trained Stable Diffusion, plus a CLIP-based Plugin-Selector that picks plugins from text instructions, for eight low-level vision tasks.

Problem

Pre-trained diffusion models suffer from randomness in the diffusion process, causing content distortion when used for low-level vision tasks that require detail preservation, and they cannot schedule multiple tasks from text.

Approach

A lightweight Task-Plugin with a Task-Prompt Branch (CLIP visual prompts distilled into cross-attention guidance) and a Spatial Complement Branch (VAE features injected as residual at the decoder's last stage) is trained per task on a frozen Stable Diffusion; a contrastive-trained Plugin-Selector maps text instructions to plugins.

Key results

• Diff-Plugin achieves the best FID/KID among compared diffusion-based methods on most tasks, e.g., desnowing FID 34.30 vs. SD 35.24 and ControlNet 34.36.
• In a 46-participant user study, Diff-Plugin has the best average ranking (1.17) vs. 2.75 for the best regression-based competitor.
• Plugin-Selector reaches mAP 0.936 on 120,000 multi-task samples; Task-Plugin parameters are only 1.67% of SD.

Measured effects reported in the paper

Surprising findings

• Regression-based specialized models and multi-task methods remain better on sparse degradation tasks such as demoireing and highlight removal.
• Removing the visual projection head causes training collapse with NaN outputs in the Plugin-Selector.

Limitations

• Cannot perform local editing; may remove degradation across the whole image rather than specific regions.
• Slightly inferior to regression-based specialized models on sparse degradation tasks like demoireing and highlight removal.

Experimental setup

Datasets: Snow100K Reside Gopro FFHQ LOL LCDMoire SHIQ merged train deraining

Benchmarks: Realistic RTTS RealBlur-J real test LFW merged low-light LCDMoire SHIQ

Metrics: FID KID mAP ZTA user ranking (average ranking)

Baselines: PromptIR AirNet WGWS-Net Restormer ControlNet InstructP2P PNP Null-Text P2P SD img2img

Builds on: Stable Diffusion CLIP DDIM Inversion ControlNet

Closest papers by meaning

• Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision (ICLR 2025 · similarity 0.89)
• Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.89)
• Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models (ICLR 2025 · similarity 0.89)
• PromptFix: You Prompt and We Fix the Photo (NeurIPS 2024 · similarity 0.89)
• DiTFastAttn: Attention Compression for Diffusion Transformer Models (NeurIPS 2024 · similarity 0.88)
• DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Data (CVPR 2024 · similarity 0.88)