Research map / Image generation / Image restoration
InstructRestore: Region-Customized Image Restoration with Human Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstructRestore, a ControlNet-based diffusion restoration model that follows text instructions to restore user-specified regions at adjustable fidelity scales, trained on a 536,945-triplet auto-annotated dataset.
Problem
Existing diffusion prior-based image restoration applies uniform processing to the whole image and cannot customize restoration intensity per region or preserve bokeh according to user instructions.
Approach
Use ControlNet cross-attention features to predict a target-region mask, then modulate ControlNet conditional features with separate scales inside vs. outside the mask, driven by parsed user instructions (fidelity scales) during inference.
Key results
- On Instruct100Set target area, PSNR varied by 1.02 dB across fidelity scales 0.5 to 1.1 while surrounding area varied only 0.29 dB, indicating localized control.
- On the bokeh test set, InstructRestore achieves full-image PSNR 24.69 and LPIPS 0.3394, best among compared methods (PASD 24.27 / 0.3523).
- Target-area MUSIQ 68.17 and MANIQA 0.6137 are the highest among compared methods on Instruct100Set, but target-area PSNR (30.55) trails PASD (31.80).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase target-region fidelity scale from 0.5 to 1.1 (ablation) | InstructRestore · Instruct100Set, target area | PSNR: 29.71 -> 30.73 | Helped |
| Increase target-region fidelity scale from 0.5 to 1.1 (ablation) | InstructRestore · Instruct100Set, target area | CLIPIQA: 0.6801 -> 0.6659 | Hurt |
| Increase target-region fidelity scale from 0.5 to 1.1 (ablation) | InstructRestore · Instruct100Set, target area | MUSIQ: 67.86 -> 66.92 | Hurt |
| Increase target-region fidelity scale from 0.5 to 1.1 (ablation) | InstructRestore · Instruct100Set, target area | MANIQA: 0.6108 -> 0.5934 | Hurt |
| Vary target-region fidelity scale while keeping surrounding-area scale at 1 (ablation) | InstructRestore · Instruct100Set, remaining area | PSNR: 31.27 -> 31.56 | No clear effect |
| Use region-specific bokeh and foreground control | InstructRestore · Bokeh testset | Bokeh IoU: 0.8234 -> 0.8482 | Helped |
Surprising findings
- Target-area PSNR/SSIM of the proposed method is lower than PASD and RealESRGAN on Instruct100Set, so no single metric wins uniformly.
Limitations
- No instance-level object specification; requires instance-level masks and captions.
- Users must follow a predefined instruction template.
- Focuses on localized customization; global quality optimization is less explored.
Experimental setup
Datasets: LSDIR EntitySeg EBB! bokeh dataset RealSR DRealSR RAIM challenge Pixabay
Benchmarks: Instruct100Set EBB! bokeh test set (70 images)
Metrics: PSNR SSIM LPIPS MANIQA MUSIQ CLIPIQA Bokeh IoU
Baselines: RealESRGAN StableSR DiffBIR PASD SeeSR SUPIR OSEDiff
Builds on: ControlNet Stable Diffusion Real-ESRGAN degradation OSEDiff
Closest papers by meaning
- Boosting Image Restoration via Priors from Pre-trained Models (CVPR 2024 · similarity 0.87)
- Adapting Text-to-Image Generation with Feature Difference Instruction for Generic Image Restoration (CVPR 2025 · similarity 0.87)
- Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (CVPR 2024 · similarity 0.86)
- Visual-Instructed Degradation Diffusion for All-in-One Image Restoration (CVPR 2025 · similarity 0.86)
- Intra and Inter Parser-Prompted Transformers for Effective Image Restoration (AAAI 2025 · similarity 0.86)
- Dual Prompting Image Restoration with Diffusion Transformers (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.