Research map / Vision-language and multimodal / Human and embodied modeling
Sentence-level Prompts Benefit Composed Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPRC, which uses a BLIP-2 Q-Former to generate a sentence-level prompt from the reference image and relative caption, concatenates it with the caption, and retrieves with BLIP-2, trained with image-text contrastive and text prompt alignment losses.
Problem
Composed image retrieval (CIR) struggles when the target image requires complex changes to the reference image, such as object removal or attribute modification, because late-fusion and pseudo-word token methods cannot decouple multiple objects.
Approach
Instead of pseudo-word tokens or late fusion, generate a dynamic sentence-level prompt (32 tokens) from both the reference image and relative caption using a Q-Former on BLIP-2, concatenate it with the caption, and reuse a text-to-image retrieval model; add an image-text contrastive loss and a text prompt alignment loss toward an optimization-derived auxiliary prompt.
Key results
- On Fashion-IQ validation, SPRC reaches 64.85 average recall versus 62.15 for Re-ranking; the two-stage SPRC² variant is the best reported.
- On CIRR test, SPRC with BLIP-2 reaches Recall@1 of 51.96 versus 40.15 for BLIP4CIR+Bi, and Avg of 81.39 versus 80.90 for Re-ranking.
- Ablation with BLIP-2 retrieval shows Ours+BLIP-2 reaches 83.02 Avg on CIRR, against 78.22 for BLIP-2-4CIR (Sum).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sentence-level prompts conditioned on both the reference image and relative caption instead of a fixed text prompt | CIR retrieval · CIRR | Recall@1: 40.15 -> 51.96 | Helped |
| Add Re-ranking as a second stage (ablation) | SPRC · CIRR | Avg.: 81.39 -> 82.66 | Helped |
| Replace CLIP retrieval with BLIP-2 retrieval (ablation) | SPRC · CIRR | Avg.: 73.33 -> 83.02 | Helped |
| Replace BLIP retrieval with BLIP-2 retrieval (ablation) | SPRC · CIRR | Avg.: 78.70 -> 83.02 | Helped |
| Generate prompts from both the reference image and caption rather than caption only (ablation) | SPRC · CIRR | Avg.: 71.29 -> 83.02 | Helped |
| Generate prompts from both the reference image and caption rather than image only (ablation) | SPRC · CIRR | Avg.: 79.72 -> 83.02 | Helped |
| Set prompt length to 32 rather than 16 (ablation) | SPRC · CIRR | Avg.: 82.97 -> 83.02 | Helped |
| Set alignment-loss weight γ to 0.6 rather than 0.8 (ablation) | SPRC · CIRR | Avg.: 82.92 -> 83.02 | Helped |
| Set alignment-loss weight γ to 0.8 rather than 0.4 (ablation) | SPRC · Fashion-IQ | Avg.: 64.56 -> 64.85 | Helped |
| Increase prompt length from 4 to 16 (ablation) | SPRC · CIRR | Avg.: 81.13 -> 82.97 | Helped |
| Increase prompt length from 4 to 32 (ablation) | SPRC · Fashion-IQ | Avg.: 62.43 -> 64.85 | Helped |
| Increase alignment-loss weight beyond γ=0.8 (ablation) | SPRC · CIRR and Fashion-IQ | Avg. recall | Hurt |
Surprising findings
- Adding view-angle changes to the caption barely changes BLIP-2 similarity, so the method fails on view-angle edits.
Limitations
- Fails on changes in object view angle, which BLIP-2 knowledge does not capture.
- The CIRR split is a random 8:1:1 re-split rather than the official protocol.
Experimental setup
Datasets: Fashion-IQ CIRR
Benchmarks: Fashion-IQ CIRR
Metrics: Recall@K Recall_subset@K
Baselines: Re-ranking BLIP4CIR+Bi CompoDiff CASE CoVR-BLIP LF-CLIP (Combiner) Pic2Word TG-CIR
Builds on: BLIP-2 Pic2Word BLIP4CIR+Bi CLIP4CIR
Closest papers by meaning
- ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval (CVPR 2025 · similarity 0.91)
- Generative Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.90)
- Do Composed Image Retrieval Benchmarks Require Multimodal Composition? (NeurIPS 2026 · similarity 0.89)
- Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR 2024 · similarity 0.89)
- Leveraging Large Vision-Language Model as User Intent-Aware Encoder for Composed Image Retrieval (AAAI 2025 · similarity 0.89)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.