Research map / Image generation / Text-to-image diffusion
Brush2Prompt: Contextual Prompt Generator for Object Inpainting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Brush2Prompt trains a CLIP-space diffusion prior that maps masked-image embeddings to CLIP-text embeddings (CatDiff/CapDiff) plus decoders, suggesting diverse, context- and mask-aware prompts for object inpainting without user text.
Problem
Users must write text prompts to insert objects via text-guided inpainting, and empty or vague prompts default to dominant background-like content; there is no way to suggest contextually plausible and diverse object prompts from an image and mask.
Approach
Reformulate object-prompt suggestion as a stochastic generative problem: a diffusion prior in CLIP space predicts CLIP-text embeddings from global-local CLIP patch embeddings of the masked image, with mask-shape augmentation for mask awareness, plus a classifier (CatDiff) or GPT-2 decoder (CapDiff) to turn embeddings into text.
Key results
- CatDiff with tight masks reaches K-5 accuracy 0.864 and K-50 entropy 0.815 on COCO, versus 0.847 accuracy and 0.043 entropy for the deterministic model without diffusion.
- CapDiff on OpenImages with tight masks scores BLEU 0.177, ROUGE 0.427, BERTScore 0.732, outperforming InstructBLIP (0.071, 0.308, 0.704) and LLaVA-5-Prompt (0.023, 0.269, 0.656) on accuracy metrics.
- CapDiff has higher sentence-level diversity (Div-4 0.858 vs 0.785 for LLaVA-5-Prompt on tight masks) but lower word-level diversity (Dist-1 0.845 vs 0.998).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CapDiff instead of generic visual-language models | Object prompt generation · OpenImages Brush2PromptBench, tight mask | BLEU: 0.071 -> 0.177 | Helped |
| use CapDiff instead of generic visual-language models | Object prompt generation · OpenImages Brush2PromptBench, BBox | BERTScore: 0.691 -> 0.715 | Helped |
| use CapDiff instead of generic visual-language models | Object prompt generation · OpenImages Brush2PromptBench, tight mask | Dist-1: 0.998 -> 0.845 | Hurt |
| use both global and local image embeddings instead of global embeddings only (ablation) | CatDiff · COCO classification, regular mask | K-1 accuracy: 0.681 -> 0.700 | Helped |
| use both global and local image embeddings instead of local embeddings only (ablation) | CatDiff · COCO classification, regular mask | K-1 accuracy: 0.681 -> 0.700 | Helped |
| combine global and local image embeddings (ablation) | CatDiff · COCO classification, regular mask | K-50 entropy: 0.884 -> 0.815; 0.845 -> 0.815 | Mixed |
| add the CatDiff diffusion prior (ablation) | Category prediction · COCO classification, tight mask | K-50 entropy: 0.043 -> 0.815 | Helped |
| add the CatDiff diffusion prior (ablation) | Category prediction · COCO classification, tight mask | K-1 accuracy: 0.847 -> 0.700 | Hurt |
| add the CatDiff diffusion prior (ablation) | Category prediction · COCO classification, tight mask | K-5 accuracy: 0.847 -> 0.864 | Helped |
| add the CapDiff diffusion prior (ablation) | Caption generation · OpenImages Brush2PromptBench, tight mask | Div-4: 0.808 -> 0.858 | Helped |
| add the CapDiff diffusion prior (ablation) | Caption generation · OpenImages Brush2PromptBench, tight mask | BLEU: 0.249 -> 0.177 | Hurt |
| add the CapDiff diffusion prior (ablation) | Caption generation · OpenImages Brush2PromptBench, tight mask | Self-BLEU: 0.215 -> 0.169 | Helped |
Surprising findings
- Adding the diffusion prior lowers caption alignment scores (BLEU, BERTScore) relative to the deterministic decoder, a trade-off against diversity.
- Generic VLMs such as InstructBLIP and LLaVA score lower than the task-specific model on caption metrics, despite their broad training.
Limitations
- Limited vocabulary depth and word-level diversity (Dist-1 is low), attributed to training dataset size.
- Diffusion prior reduces alignment with ground-truth captions, trading accuracy for diversity.
- Caption ground truth comes from BLIP captions with a label-match filter, so evaluation labels are noisy.
Experimental setup
Datasets: COCO OpenImages
Benchmarks: COCO classification Brush2PromptBench (OpenImages test subset)
Metrics: K-1 accuracy K-5 accuracy K-50 entropy BLEU ROUGE BERTScore Dist-1 Self-BLEU Div-4
Baselines: BLIP-VQA InstructBLIP LLaVA-Resample LLaVA-5-Prompt deterministic transformer (no CatDiff) CapDiff without diffusion prior
Builds on: DALL-E 2 diffusion prior CLIP DDIM ClipCap-style prefix decoding SmartBrush
Closest papers by meaning
- HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models (ICLR 2025 · similarity 0.87)
- Paint by Inpaint: Learning to Add Image Objects by Removing Them First (CVPR 2025 · similarity 0.87)
- MTADiffusion: Mask Text Alignment Diffusion Model for Object Inpainting (CVPR 2025 · similarity 0.86)
- Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision (ICLR 2025 · similarity 0.86)
- Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency (CVPR 2025 · similarity 0.86)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.