Research map / Vision-language and multimodal / CLIP and zero-shot
Vision-by-Language for Training-Free Compositional Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CIReVL, a training-free zero-shot compositional image retrieval pipeline that captions the reference image with a pretrained VLM (BLIP-2), recomposes the caption with an LLM given the modification text, and retrieves with CLIP.
Problem
Compositional image retrieval (CIR) requires an image plus a textual modification to retrieve a target image; existing zero-shot CIR methods still train textual-inversion mapping networks on large image-caption datasets.
Approach
Replace trained textual inversion with off-the-shelf generative captioning (BLIP-2) and LLM-based language reasoning that rewrites the caption according to the modifier, then retrieve with CLIP text-to-image matching; the pipeline is modular, training-free and human-readable so captions can be intervened on.
Key results
- On CIRCO with ViT-L/14, CIReVL reaches mAP@5 18.57% vs 11.68% for SEARLE (best trained baseline).
- With OpenCLIP ViT-G/14 retrieval, CIReVL reaches CIRCO mAP@5 26.77% and CIRR R@1 34.64% after scaling the retrieval model.
- On GeneCIS, CIReVL averages 16.8% R@1 with ViT-G/14 (about 15.9% for Combiner trained on CC3M), surpassing the CIRR-finetuned Combiner.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CIReVL instead of SEARLE | ViT-L/14 retrieval on CIRCO · CIRCO test | mAP@5: 11.68 -> 18.57 | Helped |
| use CIReVL instead of SEARLE | ViT-L/14 retrieval on CIRR · CIRR test | Recall@1: 24.24 -> 24.55 | Helped |
| use CIReVL instead of SEARLE | ViT-B/32 retrieval on CIRR · CIRR test | R_s@1: 54.89 -> 60.17 | Helped |
| increase retrieval backbone from ViT-L/14 to ViT-G/14 | CIReVL · CIRCO test | mAP@5: 18.57 -> 26.77 | Helped |
| increase retrieval backbone from ViT-L/14 to ViT-G/14 | CIReVL · CIRR test | Recall@1: 24.55 -> 34.65 | Helped |
| replace GPT-3.5-Turbo with GPT-4 (ablation) | CIReVL with BLIP-2 and ViT-B/32 · CIRCO validation | mAP@5: 13.33 -> 15.63 | Helped |
| replace GPT-3.5-Turbo with Vicuna-13B (ablation) | CIReVL with BLIP-2 and ViT-B/32 · CIRCO validation | mAP@5: 13.33 -> 12.65 | Hurt |
| replace GPT-3.5-Turbo with Llama2-70B (ablation) | CIReVL with BLIP-2 and ViT-B/32 · CIRCO validation | mAP@5: 13.33 -> 10.22 | Hurt |
| remove LLM reasoning and use a caption template (ablation) | CIReVL with BLIP-2 and ViT-B/32 · CIRCO validation | mAP@5: 13.33 -> 9.22 | Hurt |
| replace BLIP-2 with CoCa (ablation) | CIReVL with GPT-3.5-Turbo and ViT-B/32 · CIRCO validation | mAP@5: 13.33 -> 13.64 | No clear effect |
| replace BLIP-2 with BLIP (ablation) | CIReVL with GPT-3.5-Turbo and ViT-B/32 · CIRCO validation | mAP@5: 13.33 -> 13.1 | No clear effect |
| use CIReVL instead of SEARLE | ViT-L/14 retrieval on FashionIQ · FashionIQ validation average | Recall@10: 25.56 -> 28.55 | Helped |
Surprising findings
- A simple captioning-only template baseline reaches mAP@5 9.22% on CIRCO validation, and CLIP retrieval, not reasoning, is identified as the main bottleneck.
- Scaling the CLIP retrieval model yields log-linear gains on CIRCO and CIRR, contrasting with prior work that observed only minimal gains from scaling.
Limitations
- Retrieval via CLIP is a bottleneck; LLM-generated targets may be correct but not retrieved.
- Benchmarks such as CIRR are noisy and annotations incomplete, so some correct retrievals are penalized.
- Performance of public LLMs (Vicuna-13B, Llama2-70B) lags GPT-3.5/GPT-4.
Experimental setup
Datasets: CIRR CIRCO FashionIQ GeneCIS ImageNet ImageNet-R
Benchmarks: CIRCO CIRR FashionIQ GeneCIS ImageNet domain conversion
Metrics: mAP@k Recall@k R_s@k TIFA
Baselines: Image-only Text-only Image + Text PALAVRA Pic2Word SEARLE Combiner
Builds on: CLIP textual inversion Pic2Word SEARLE Vision-by-language
Closest papers by meaning
- Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.92)
- CoLLM: A Large Language Model for Composed Image Retrieval (CVPR 2025 · similarity 0.92)
- ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval (CVPR 2025 · similarity 0.92)
- Generative Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.91)
- Language-only Training of Zero-shot Composed Image Retrieval (CVPR 2024 · similarity 0.91)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.