Research map / Vision-language and multimodal / CLIP and zero-shot
Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds COCO-Facet, a 9,112-query benchmark for attribute-focused text-to-image retrieval, and shows that GPT-prompted, promptable image embeddings from MLLM-based VLM2Vec improve retrieval, with pre-processing and linear-approximation accelerations.
Problem
Current text-to-image retrievers (CLIP-like and MLLM-based embedders) struggle to retrieve images based on specific, often non-dominant visual attributes such as time, weather, material, or people count, since general image embeddings focus on global semantics and subjects.
Approach
Condition image embeddings on attribute-specific textual prompts (GPT-4o-generated questions) using MLLM-based universal embedders so the embedding highlights the queried attribute; add two acceleration strategies (pre-computed prompt-specific embeddings and a test-time linear approximation of the embedder).
Key results
- Promptable (GPT-prompt) VLM2Vec-Phi-3.5-V raises average COCO-Facet Recall@5 from 58.9 to 75.5 and Recall@1 from 44.5 to 53.4.
- Pre-processed promptable embeddings with GPT-4o prompt selection reach 73.7 average Recall@5 (vs 58.9 baseline), a ~15-point gain.
- Test-time linear approximation with K=100 raises average Recall@5 to 67.0 from 58.9, though Recall@1 drops to 42.5 from 44.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Condition image embeddings on category-specific GPT prompts | VLM2Vec-Phi-3.5-V · COCO-Facet | Recall@5: 58.9 -> 75.5 | Helped |
| Condition image embeddings on category-specific GPT prompts | VLM2Vec-Phi-3.5-V · COCO-Facet | Recall@1: 44.5 -> 53.4 | Helped |
| Condition image embeddings on category-specific GPT prompts | VLM2Vec-Phi-3.5-V · Place365 | Recall@5: 58.2 -> 66.7 | Helped |
| Condition image embeddings on category-specific GPT prompts | VLM2Vec-Phi-3.5-V · SUN397 | Recall@5: 85.2 -> 88.9 | Helped |
| Select a predefined GPT prompt at test time (ablation) | VLM2Vec-Phi-3.5-V · COCO-Facet | Recall@5: 58.9 -> 73.7 | Helped |
| Approximate promptable embeddings with a test-time linear mapping (K=100) (ablation) | VLM2Vec-Phi-3.5-V · COCO-Facet | Recall@5: 58.9 -> 67.0 | Helped |
| Use text embeddings of MLLM-generated answers as target embeddings instead of promptable image embeddings (ablation) | VLM2Vec-Phi-3.5-V retrieval pipeline · COCO-Facet | Recall@1: 53.4 -> 40.5 | Hurt |
| Use human-written rather than GPT-written prompts (ablation) | VLM2Vec-Phi-3.5-V · COCO-Facet | Recall@1: 44.5 -> 56.3 | Helped |
| Replace query wording with CLIP-style text (ablation) | CLIP-ViT-L/14 · COCO-Facet | Recall@1: 33.7 -> 33.8 | No clear effect |
| Replace query wording with CLIP-style text (ablation) | CLIP-ViT-L/14 · COCO-Facet | Recall@5: 47.0 -> 47.6 | No clear effect |
| Use the promptable embedding pipeline rather than general image embeddings | VLM2Vec-Phi-3.5-V · COCO-Facet Animals | Recall@1: 95.5 -> 90.7 | Hurt |
Surprising findings
- Larger output dimension and MLLM-based scale do not resolve the attribute gap: MLLM embedders still have Recall@1 below 10% on some attributes.
- A text-based approach (LLaVA-1.5 answer embedded with GRIT) underperforms promptable embeddings and even falls below the no-prompt baseline in Recall@1, due to hallucination and linguistic ambiguity.
Limitations
- Linear approximation fails to match the no-prompt baseline on the common attributes Animals, Scenes, and Objects.
- Pre-processing requires storing prompt-specific embeddings for each predefined prompt, adding memory and pre-processing cost.
- Gesture category selection accuracy is low (8.9%) due to prompt similarity with Count of People.
Experimental setup
Datasets: MSCOCO 2017 COCO-Stuff Visual7W VisDial Place365 SUN397
Benchmarks: COCO-Facet MSCOCO 2017 validation Place365 (converted) SUN397 (converted)
Metrics: Recall@1 Recall@5
Baselines: CLIP-ViT-L/14 EVA01 ViT-g-14 EVA02 ViT-bigE-14+ SigLIP ViT-SO-14 SigLIP2 ViT-SO-14 BLIP2-COCO MagicLens E5-V MM-Embed MMRet-MLLM-S2 LLaVE-2B VLM2Vec-Phi-3.5-V
Builds on: VLM2Vec CLIP MSCOCO Visual7W COCO-Stuff
Closest papers by meaning
- NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.86)
- MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions (ICML 2024 · similarity 0.86)
- Sentence-level Prompts Benefit Composed Image Retrieval (ICLR 2024 · similarity 0.86)
- Preserving Clusters in Prompt Learning for Unsupervised Domain Adaptation (CVPR 2025 · similarity 0.86)
- Prompt Tuning In a Compact Attribute Space (AAAI 2025 · similarity 0.86)
- Context-I2W: Mapping Images to Context-Dependent Words for Accurate Zero-Shot Composed Image Retrieval (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.