Research map / Vision-language and multimodal / CLIP and zero-shot
Language-only Training of Zero-shot Composed Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LinCIR, a language-only training method for zero-shot composed image retrieval that learns a text-to-token projection via self-masking projection and noise augmentation, trained only on captions.
Problem
Zero-shot composed image retrieval (ZS-CIR) methods rely on image inputs and a restricted prompt ("a photo of [$]"), which limits backbone scalability, generalization to diverse text conditions, and training efficiency.
Approach
Train a projection module from the text latent space to the token embedding space using only text, via Self-Masking Projection (replacing keyword tokens with the projected embedding and matching latents with MSE), plus a non-Gaussian random noise (N(0,1)×Unif(0,1)) to reduce the modality gap at inference on image embeddings.
Key results
- With CLIP ViT-G backbone, LinCIR reaches 19.71 mAP@5 on CIRCO versus 13.20 for SEARLE and 5.54 for Pic2Word.
- On FashionIQ, LinCIR ViT-G achieves 45.11 average R@10, exceeding the supervised Combiner baseline's 38.32.
- LinCIR with ViT-G trains in 48 minutes on 8 A100 GPUs, about 16.4x faster than Pic2Word-style training at that scale.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use language-only LinCIR instead of SEARLE | CLIP ViT-G zero-shot CIR · CIRCO | mAP@5: 13.20 -> 19.71 | Helped |
| Use language-only LinCIR instead of SEARLE | CLIP ViT-G zero-shot CIR · GeneCIS | R@3: 32.46 -> 33.54 | Helped |
| Use language-only LinCIR instead of the supervised Combiner | FashionIQ retrieval · FashionIQ average | R@10: 38.32 -> 45.11 | Helped |
| Use language-only LinCIR instead of SEARLE | CLIP ViT-G zero-shot CIR · CIRR full | R@10: 75.11 -> 76.05 | Helped |
| Use language-only LinCIR instead of Pic2Word | CLIP ViT-G zero-shot CIR · CIRR subset | R@1: 68.92 -> 63.35 | Hurt |
| Replace keyword masking with the prompt “a photo of [$]” (ablation) | LinCIR projection training · CIRR dev | R@1: 25.66 -> 21.65 | Hurt |
| Replace keyword masking with the prompt “a photo of [$]” (ablation) | LinCIR projection training · FashionIQ | R@50: 46.49 -> 44.35 | Hurt |
| Use visual rather than textual embeddings before the projection (ablation) | LinCIR projection training · CIRR dev | R@1: 25.66 -> 22.63 | Hurt |
| Mask all non-keyword tokens instead of all keywords (ablation) | LinCIR SMP · CIRR dev | R@1: 25.66 -> 20.59 | Hurt |
| Mask all noun tokens instead of adjective-and-noun keywords (ablation) | LinCIR SMP · CIRR dev | R@1: 25.66 -> 24.95 | Hurt |
| Remove noise addition (ablation) | LinCIR training · CIRR dev | R@1: 25.47 -> 19.76 | Hurt |
| Use Gaussian noise N(0, 1) instead of the selected noise distribution (ablation) | LinCIR training · CIRR dev | R@1: 25.47 -> 23.70 | Hurt |
Surprising findings
- Pic2Word and SEARLE degrade when scaled from ViT-H to ViT-G, while LinCIR keeps improving.
- Training with paired image-text data (CC3M images) performed worse than text-only SMP training, which the authors attribute to overfitting to CC3M image-text relationships.
Limitations
- R@1 metrics on CIRR, FashionIQ and GeneCIS are noisy due to false negatives, so the authors emphasize mAP and R@10 instead.
- CIRR results are mixed; LinCIR is best only on full R@10 and not on other metrics.
Experimental setup
Datasets: CC3M StableDiffusion prompts (2.47M) CIRCO GeneCIS FashionIQ CIRR
Benchmarks: CIRCO GeneCIS FashionIQ CIRR
Metrics: mAP@K Recall@K
Baselines: Pic2Word SEARLE Combiner
Builds on: Pic2Word SEARLE CLIP Nukrai et al. noise injection for captioning
Closest papers by meaning
- ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval (CVPR 2025 · similarity 0.92)
- Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.91)
- Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR 2024 · similarity 0.91)
- Generative Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.91)
- Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval (ICLR 2024 · similarity 0.91)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.