Research map / Image generation / Image restoration
Dual Prompting Image Restoration with Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DPIR, a diffusion-transformer (SD3-based) image restoration method that injects low-quality image features via a lightweight ControlNeXt-style branch and adds visual-text dual prompts from CLIP global-local features plus T5 text.
Problem
Conditioning diffusion transformers (DiTs) for image restoration is underexplored; existing control methods designed for U-Net latent diffusion models do not suit DiTs, which lack skip connections and lose LQ input information.
Approach
Combine a lightweight convolutional LQ conditioning branch (adaptive mean/variance alignment into the first DiT block) with a dual prompting module that replaces CLIP text embeddings by CLIP visual features of local and surrounding global patches, alongside T5 text, trained with a global-local visual strategy.
Key results
- On DRealSR synthetic evaluation, DPIR reaches DISTS 0.1998 and CLIPIQA 0.6424, the best or second best among compared methods, with PSNR 27.31 below Real-ESRGAN and StableSR.
- On real-world DRealSR, DPIR achieves the best no-reference scores: CLIPIQA 0.6980, MUSIQ 70.85, NIQE 3.7382, MANIQA 0.5310.
- Ablation: global-local visual prompts improve DIV2K-Val MANIQA from 0.4315 to 0.5330 and CLIPIQA from 0.6730 to 0.7416 over local-only training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace text-only prompting with visual-text dual prompts (ablation) | DPIR · DRealSR synthetic | LPIPS: 0.4217 -> 0.3846 | Helped |
| replace text-only prompting with visual-text dual prompts (ablation) | DPIR · DRealSR synthetic | CLIPIQA: 0.5614 -> 0.6175 | Helped |
| remove T5 text prompts and use visual prompts only (ablation) | DPIR · DRealSR synthetic | LPIPS: 0.3846 -> 0.6239 | Hurt |
| remove T5 text prompts and use visual prompts only (ablation) | DPIR · DRealSR synthetic | CLIPIQA: 0.6175 -> 0.4106 | Hurt |
| add global visual context to local visual-prompt training (ablation) | DPIR · DIV2K-Val | LPIPS: 0.3622 -> 0.3284 | Helped |
| add global visual context to local visual-prompt training (ablation) | DPIR · DIV2K-Val | CLIPIQA: 0.6730 -> 0.7416 | Helped |
| use DPIR rather than SinSR | image restoration · RealSR synthetic | CLIPIQA: 0.6678 -> 0.6625 | Hurt |
| use DPIR rather than SinSR | image restoration · CelebA synthetic | CLIPIQA: 0.7940 -> 0.7111 | Hurt |
| use DPIR rather than SinSR | image restoration · DRealSR real-world | CLIPIQA: 0.6842 -> 0.6980 | Helped |
| use DPIR rather than StableSR | image restoration · DRealSR real-world | MUSIQ: 68.58 -> 70.85 | Helped |
Surprising findings
- Visual-prompts-only variant gets the highest PSNR (28.69) but much worse perceptual metrics (LPIPS 0.6239, NIQE 12.64), and the dual prompt does not beat it in PSNR.
- DPIR does not achieve the best PSNR on most synthetic datasets, trading fidelity metrics for no-reference perceptual quality.
Limitations
- Visual-prompts-only variant performs much worse, indicating reliance on text for global semantics
- Training relies on a large self-collected 20M-image dataset that is not released
- Lower PSNR than several baselines on synthetic benchmarks
Experimental setup
Datasets: self-collected dataset (over 20M images) DIV2K DRealSR RealSR CelebA
Benchmarks: DIV2K-Val DRealSR RealSR CelebA
Metrics: PSNR SSIM LPIPS DISTS CLIPIQA MUSIQ NIQE MANIQA
Baselines: BSRGAN Real-ESRGAN StableSR SinSR SUPIR DACLIP
Builds on: SD3 ControlNeXt SUPIR Real-ESRGAN
Closest papers by meaning
- UP-Restorer: When Unrolling Meets Prompts for Unified Image Restoration (AAAI 2025 · similarity 0.87)
- CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation (CVPR 2024 · similarity 0.87)
- Learning Diffusion Texture Priors for Image Restoration (CVPR 2024 · similarity 0.87)
- Residual Denoising Diffusion Models (CVPR 2024 · similarity 0.86)
- InstructRestore: Region-Customized Image Restoration with Human Instructions (NeurIPS 2025 · similarity 0.86)
- DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.