Research map / Vision-language and multimodal / Visual question answering
Unifying Image Processing as Visual Prompting Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PromptGIP, a ViT-large visual prompting QA model that unifies up to 15 image processing tasks (restoration, enhancement, edge detection) by encoding input-output image pairs as Q-A-Q-A sequences with masked answer prediction.
Problem
General image processing (restoration, enhancement, feature extraction with distinct output domains) usually needs task-specific models; existing visual in-context learning methods focus on high-level vision and handle low-level tasks poorly.
Approach
Treat input-output image pairs as question-answer prompts in a Q-A-Q-A sequence, with random masking of answer images during training, so a single ViT reprograms image processing as visual prompting QA and handles tasks with different output domains without finetuning.
Key results
- PromptGIP (ViT-large, all-tasks) reaches 26.22/0.8167 PSNR/SSIM on Gaussian noise vs Painter's 24.17/0.7468 in the all-in-one setting.
- PromptGIP beats Painter on LLE (20.30 vs 19.47 PSNR) and LLF (26.11 vs 23.87 PSNR), and on Canny MAE (21.44 vs 33.72).
- PromptGIP outperforms Restormer (all-task variant) on complex rain (24.08 vs 20.39) and haze (24.32 vs 7.22) but trails Restormer on most other restoration tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use visual prompting QA with Q-A-Q-A encoding rather than Painter's Q-Q-A-A encoding (ablation) | PromptGIP · Poisson Noise | PSNR: 24.63 -> 27.29 | Helped |
| Use visual prompting QA with Q-A-Q-A encoding rather than Painter's Q-Q-A-A encoding (ablation) | PromptGIP · Haze | PSNR: 20.60 -> 24.32 | Helped |
| Use visual prompting QA with Q-A-Q-A encoding rather than Painter's Q-Q-A-A encoding (ablation) | PromptGIP · LLF | PSNR: 23.87 -> 26.11 | Helped |
| Use visual prompting QA with Q-A-Q-A encoding rather than Painter's Q-Q-A-A encoding (ablation) | PromptGIP · Laplacian | MAE: 5.4518 -> 3.7852 | Helped |
| Mask both answer images rather than only A2 (ablation) | PromptGIP · Poisson Noise | PSNR: 26.30 -> 27.29 | Helped |
| Mask both answer images rather than only A2 (ablation) | PromptGIP · Haze | PSNR: 18.57 -> 24.32 | Helped |
| Mask both answer images rather than only A2 (ablation) | PromptGIP · LLF | PSNR: 25.38 -> 26.11 | Helped |
| Mask both answer images rather than only A2 (ablation) | PromptGIP · Laplacian | MAE: 11.5553 -> 3.7852 | Helped |
| Use PromptGIP rather than Restormer | image restoration models · Complex Rain | PSNR: 24.26 -> 24.08 | Hurt |
| Use PromptGIP rather than Restormer | image restoration models · Haze | PSNR: 14.83 -> 24.32 | Hurt |
| Use PromptGIP rather than Painter | image processing models · LLE (LOL dataset) | PSNR: 19.47 -> 20.30 | Helped |
| Use PromptGIP rather than Painter | image processing models · Canny | MAE: 33.7188 -> 21.4376 | Helped |
Surprising findings
- Painter's visual prompts often fail: with denoising prompts on indoor images it performs depth estimation instead, suggesting dataset memorization.
- The model fails to perform colorization from a grayscale-color prompt pair, so out-of-domain generalization is limited.
Limitations
- Plain ViT backbone with 16x16 patching loses high-frequency detail, so it underperforms Restormer on most restoration tasks.
- Limited training data and compute prevent large-scale scaling experiments; no emergent behavior beyond training tasks.
- Prompt selection matters: 20 prompts per task are tried and the best is reported, with noticeable variance across prompts.
Experimental setup
Datasets: ImageNet RESIDE ITS Rain13K LOL MIT-Adobe FiveK Set5 Set14 BSDS100 Manga109 Urban100 General100 DIV2K
Benchmarks: Common528 LOL MIT-Adobe FiveK SOTS-indoor Test100
Metrics: PSNR SSIM MAE
Baselines: Restormer Real-ESRGAN Painter ViT-large
Builds on: Painter MAE-VQGAN masked autoencoder
Closest papers by meaning
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.89)
- LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.89)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.88)
- AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.88)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.87)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.