Research map / Image generation / Text-to-image diffusion
PromptFix: You Prompt and We Fix the Photo
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PromptFix, a diffusion-based all-in-one instruction-following model for low-level image processing, trained on a ~1.01M-triplet dataset, with high-frequency guidance sampling and a VLM-generated auxiliary prompt adapter.
Problem
Instruction-guided diffusion models lack diverse instruction-following data for low-level image processing and lose high-frequency spatial detail due to VAE compression, and they fail on severely degraded inputs without descriptive prompts.
Approach
Combines a large instruction-following low-level dataset, a high-frequency guidance sampling procedure that adds a Fourier/Sobel high-pass fidelity loss on the VAE-decoded estimate with skip-connect LoRA features, and an auxiliary prompt cross-attention branch fed with VLM (LLaVA) semantic and defect descriptions.
Key results
- PromptFix achieves LPIPS 0.233 on colorization and 0.127 on watermark removal, outperforming InstructP2P, InstructDiff and MGIE in the instruction-driven diffusion comparison.
- On the 200-image multi-task benchmark, PromptFix reaches PSNR 22.05 and LPIPS 0.1519 versus 19.30 PSNR and 0.2282 LPIPS for PromptIR.
- Ablation shows the full HGS plus auxiliary prompting setup reaches LPIPS 0.1600 versus 0.1990 without HGS and auxiliary prompting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PromptFix to replace InstructP2P on Colorization | Instruction-driven image processing · Colorization | LPIPS: 0.394 -> 0.233 | Helped |
| Add PromptFix to replace InstructDiff on Object Removal | Instruction-driven image processing · Object Removal | LPIPS: 0.071 -> 0.054 | Helped |
| Add PromptFix to replace InstructDiff on Watermark Removal | Instruction-driven image processing · Watermark Removal | LPIPS: 0.247 -> 0.127 | Helped |
| Add PromptFix to replace Diff-Plugin on Low-light Enhancement | Image restoration · Low-light Enhancement | LPIPS: 0.227 -> 0.135 | Helped |
| Add PromptFix to replace Diff-Plugin on Desnow | Image restoration · Desnow | LPIPS: 0.133 -> 0.103 | Helped |
| Add PromptFix to replace InstructDiff on Dehazy | Image restoration · Dehazy | LPIPS: 0.124 -> 0.088 | Helped |
| Add PromptFix to replace InstructDiff on Super Resolution | Image restoration · Super Resolution | LPIPS: 0.233 -> 0.143 | Helped |
| Add Fourier and Sobel high-frequency guidance sampling (ablation) | PromptFix · HGS and Auxiliary Prompting | LPIPS: 0.2068 -> 0.1600 | Helped |
| Add Fourier and Sobel high-frequency guidance sampling (ablation) | PromptFix · HGS and Auxiliary Prompting | ManIQA: 0.6487 -> 0.6274 | Hurt |
| Add auxiliary prompting (ablation) | PromptFix with Fourier and Sobel HGS · HGS and Auxiliary Prompting | LPIPS: 0.1990 -> 0.1600 | Helped |
| Add auxiliary prompting (ablation) | PromptFix with Fourier and Sobel HGS · HGS and Auxiliary Prompting | ManIQA: 0.5856 -> 0.6274 | Helped |
| Use out-of-training instructions under 20 words instead of training instructions (ablation) | PromptFix · Instruction prompts | LPIPS: 0.1600 -> 0.1639 | Hurt |
Surprising findings
- Adding HGS makes restored images slightly resemble the degraded input and appear darker than without HGS, a fidelity-quality trade-off the authors acknowledge.
- Blind restoration without instructions occasionally produces text-to-image generation from the auxiliary prompt instead of image processing.
Limitations
- Blind restoration sometimes degenerates into text-to-image generation driven by the auxiliary prompt.
- HGS trades quality for fidelity, making outputs slightly closer to the degraded input and darker.
- Performance drops with long instructions (40-70 words), attributed to the long-tail instruction length in training data.
Experimental setup
Datasets: RESIDE Dense-Haze O-Haze LOL GoPro Laion-5b Mulan Snow100K CLWD
Benchmarks: LPIPS/ManIQA on seven low-level tasks (512x512) multi-task validation set of 200 images blind restoration (low-light, desnowing, dehazing)
Metrics: LPIPS ManIQA PSNR SSIM
Baselines: InstructP2P InstructDiff MGIE PromptIR AirNet Diff-Plugin InstructDiffusion
Builds on: Stable Diffusion 1.5 InstructPix2Pix EDM LLaVA
Closest papers by meaning
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision (ICLR 2025 · similarity 0.89)
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks (CVPR 2024 · similarity 0.89)
- Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis (NeurIPS 2024 · similarity 0.88)
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.