Research map / Image generation / Image restoration
Multimodal Prompt Perceiver: Empower Adaptiveness Generalizability and Fidelity for All-in-One Image Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MPerceiver, a multimodal prompt learning approach built on Stable Diffusion 2.1 with textual (CLIP-derived) and visual (VAE multiscale) prompts weighted by a degradation predictor, plus a detail refinement module, for all-in-one image restoration.
Problem
All-in-one image restoration struggles with intricate, unknown real-world degradations, including mixed degradations, and generalization to unseen tasks; direct use of Stable Diffusion also loses fine detail due to VAE compression.
Approach
A dual-branch multimodal prompt module: a CM-Adapter maps CLIP image features of LQ inputs into degradation-aware text embeddings, and an IR-Adapter injects multiscale VAE features modulated by visual prompt pools; both are dynamically weighted by degradation probabilities from a CLIP-based predictor, and a plug-in DRM passes encoder features directly to the VAE decoder for fidelity.
Key results
- On Rain1400 deraining, MPerceiver reaches 33.40 PSNR / 0.937 SSIM vs 33.27 for NAFNet-all-in-one and 33.68 for task-specific Restormer, and it is best among all-in-one methods on 9 tasks.
- In zero-shot UDC IR on TOLED, MPerceiver achieves 32.92 PSNR vs 27.90 for the next best all-in-one method (NAFNetA); on UIEB underwater it gets 22.69 PSNR vs 17.34.
- Ablation on deraining/dehazing/raindrop average: the full model improves PSNR from 16.49 (Stable Diffusion baseline) to 29.17, with DRM adding about 4 dB over the branches alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MPerceiver instead of the strongest listed baseline | MPerceiver · Average dehazing | PSNR: 18.64 -> 20.95 | Helped |
| Use MPerceiver instead of the strongest listed baseline | MPerceiver · MID6 low-light, noise, blur | PSNR: 20.95 -> 23.84 | Helped |
| Use MPerceiver instead of the strongest listed baseline | MPerceiver · MID6 rain, raindrop, noise | PSNR: 22.72 -> 22.35 | Hurt |
| Add the textual branch (ablation) | Stable Diffusion baseline · Deraining, dehazing, raindrop removal average | PSNR: 16.49 -> 19.19 | Helped |
| Add the visual branch (ablation) | Stable Diffusion baseline · Deraining, dehazing, raindrop removal average | PSNR: 16.49 -> 25.05 | Helped |
| Add visual and textual branches (ablation) | Stable Diffusion baseline · Deraining, dehazing, raindrop removal average | PSNR: 16.49 -> 25.31 | Helped |
| Remove the textual prompt pool (ablation) | Textual branch · Deraining, dehazing, raindrop removal average | PSNR: 19.19 -> 18.94 | Hurt |
| Remove the visual prompt pool (ablation) | Visual branch · Deraining, dehazing, raindrop removal average | PSNR: 25.05 -> 24.94 | Hurt |
| Remove the textual prompt pool (ablation) | Visual and textual branches · Deraining, dehazing, raindrop removal average | PSNR: 25.31 -> 25.20 | Hurt |
| Remove the visual prompt pool (ablation) | Visual and textual branches · Deraining, dehazing, raindrop removal average | PSNR: 25.31 -> 25.13 | Hurt |
| Add the detail refinement module (ablation) | Visual and textual branches · Deraining, dehazing, raindrop removal average | PSNR: 25.31 -> 29.17 | Helped |
Surprising findings
- Restormer and NAFNet retrained in all-in-one setting are used as strong general baselines, showing that a single general restoration network remains competitive with the prompt-based approach on many tasks.
Limitations
- Degradation predictor and prompt pools assume a fixed set of known training degradation types; unseen degradations are treated only as combinations of training ones.
- Relies on SD sampling with 50 DDIM steps, which is computationally heavier than feed-forward restoration networks (not quantified in text).
Experimental setup
Datasets: Rain1400 Outdoor-Rain SSID LHP RESIDE NH-HAZE Dense-Haze Snow100K RealSnow RainDrop RainDS LOL-v2
Benchmarks: Rain1400 Dehazing (RESIDE, NH-HAZE, Dense-Haze) Snow100K-L RainDrop LOL-v2-Real GoPro DPDD CBSD68/Kodak24/McMaster/Urban100 SIDD MID6 LHP SSID
Metrics: PSNR SSIM LPIPS FID NIQE BRISQUE
Baselines: AirNet PromptIR DA-CLIP TransWeather WGWS-Net RestormerA NAFNetA Restormer DRSformer DiffIR
Builds on: Stable Diffusion CoOp CoCoOp VPT AirNet
Closest papers by meaning
- UniRestore: Unified Perceptual and Task-Oriented Image Restoration Model Using Diffusion Prior (CVPR 2025 · similarity 0.82)
- Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (CVPR 2024 · similarity 0.81)
- UP-Restorer: When Unrolling Meets Prompts for Unified Image Restoration (AAAI 2025 · similarity 0.80)
- Intra and Inter Parser-Prompted Transformers for Effective Image Restoration (AAAI 2025 · similarity 0.80)
- UARE: A Unified Vision-Language Model for Image Quality Assessment, Restoration, and Enhancement (CVPR 2026 · similarity 0.80)
- Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation (NeurIPS 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.