Research map / Image generation / Text-to-image diffusion
Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Diff-Prompt trains a Mask-VAE and a DiT-based diffusion prompt generator with mask supervision, then aligns its denoising features to a frozen GLIP-T(A) via adapters to fine-tune it for referring expression comprehension.
Problem
Prompt learning for pre-trained multimodal models gives limited gains on complex, fine-grained tasks such as referring expression comprehension because prompts are optimized directly by loss backpropagation, limiting their richness and specificity.
Approach
Use a diffusion transformer, conditioned on image and caption and supervised by ground-truth masks in a VAE latent space, to generate input-specific prompts; retain intermediate denoising latents as prompts, map them through modality adapters, and insert them with learnable global prompts into a frozen GLIP encoder.
Key results
- On RefCOCO testA, Diff-Prompt reaches 39.08 R@1 and 94.71 R@5 versus GLIP-T(A)'s 30.21 and 80.66, a maximum gain of 8.87 R@1 and 14.05 R@5.
- Diff-Prompt achieves the best R@1/R@5/UB on all three RefCOCO splits among compared adapter and prompt methods (e.g., val R@1 37.94 vs. MMA 35.28).
- On cross-dataset classification, Diff-Prompt averages 68.85, below CoCoLe (68.91) and PromptKD (71.33) and not state of the art.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Diff-Prompt instead of GLIP-T(A) | GLIP-T(A) · RefCOCO testA | R@1: 30.21 -> 39.08 | Helped |
| Use Diff-Prompt instead of GLIP-T(A) | GLIP-T(A) · RefCOCO testA | R@5: 80.66 -> 94.71 | Helped |
| Use Diff-Prompt instead of GLIP-T(A) | GLIP-T(A) · Flickr30k test | R@1: 45.57 -> 59.53 | Helped |
| Add input-specific visual prompt (ablation) | Diff-Prompt · RefCOCO val | R@1: 36.94 -> 37.94 | Helped |
| Add global visual prompt (ablation) | Diff-Prompt · RefCOCO val | R@1: 37.01 -> 37.94 | Helped |
| Add input-specific textual prompt (ablation) | Diff-Prompt · RefCOCO val | R@1: 35.61 -> 37.94 | Helped |
| Add global textual prompt (ablation) | Diff-Prompt · RefCOCO val | R@1: 36.95 -> 37.94 | Helped |
| Align denoising features to encoder layers in reverse order rather than sequentially (ablation) | Diff-Prompt · RefCOCO testA | R@1: 35.09 -> 39.08 | Helped |
| Add Diff-Prompt to CLIPSeg | CLIPSeg (PC, D=128) · PhaseCut | mIoU: 48.2 -> 49.6 | Helped |
| Add Diff-Prompt to BLIP | BLIP · VQA-v2 test-dev | accuracy: 78.24 -> 78.59 | Helped |
Surprising findings
- Zero-shot results on ODinW show Diff-Prompt underperforms GLIP-T(A) on several datasets (average AP 27.57 vs 25.73 is higher, but Pistols drops from 32.28 to 12.82), indicating uneven generalization.
- Reversing the alignment order (deeper denoising steps feeding shallow layers) substantially outperforms sequential alignment (val R@1 37.94 vs 32.27).
Limitations
- Limited to 224x224 input images due to DiT constraints.
- Multi-step diffusion sampling adds substantial computation and latency relative to other prompt methods.
- Performance depends on the generalization of the pre-trained VAE and DiT, and zero-shot gains are uneven across datasets.
Experimental setup
Datasets: RefCOCO Flickr30k ODinW PhraseCut VQA-v2 NLVR2 ImageNet
Benchmarks: RefCOCO Flickr30k ODinW PhraseCut VQA-v2 NLVR2
Metrics: R@1 R@5 UB AP mIoU IoU_FG accuracy
Baselines: GLIP-T(A) Tip-adapter Meta-adapter CLIP-Adapter MMA VPT VFPT CoOp S-Prompts MaPLe FedTPG
Builds on: DiT Latent Diffusion Models DDIM GLIP MaPLe VPT
Closest papers by meaning
- Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models (NeurIPS 2024 · similarity 0.90)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.90)
- On Discrete Prompt Optimization for Diffusion Models (ICML 2024 · similarity 0.90)
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks (CVPR 2024 · similarity 0.89)
- PromptFix: You Prompt and We Fix the Photo (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.