Research map / Vision-language and multimodal / CLIP and zero-shot
Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ISA, an asymmetric zero-shot composed image retrieval method that maps a query image to a sentence of learned tokens with an adaptive token learner, trained on unlabeled CC3M images using a frozen BLIP model.
Problem
Composed image retrieval normally needs labeled (reference, modifier, target) triplets, and symmetric zero-shot methods using large VL models are hard to deploy on resource-limited query devices.
Approach
An adaptive token learner (spatial attention plus self- and cross-attention) maps a lightweight encoder's feature map to a multi-token sentence in BLIP's word embedding space, trained with global contrastive distillation and a local alignment regularization against a frozen BLIP visual encoder and matching encoder; the query side uses a light model while the gallery side uses the large BLIP model.
Key results
- On CIRR test, EfficientNet B2-based ISA (asymmetric) reaches average 64.48 vs 59.21 for SEARLE (BLIP) and 62.38 for the symmetric variant.
- On CIRCO validation, EfficientNet B2 ISA reaches average mAP 12.74 on test-style table (14.52 on validation ablation), above SEARLE (BLIP) 11.85.
- On FashionIQ validation, EfficientNet B2 ISA averages R@10 29.60 and R@50 49.54, above SEARLE (BLIP) 27.62 and 47.56.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace single-token MLP mapping with the adaptive token learner (ablation) | ISA with EfficientNet B2 query encoder · CIRR validation | Avg Recall@K: 62.57 -> 66.33 | Helped |
| Replace single-token MLP mapping with the adaptive token learner (ablation) | ISA with EfficientNet B2 query encoder · CIRCO validation | Avg mAP@K: 9.14 -> 14.52 | Helped |
| Add local alignment regularization to GCD (ablation) | ISA with EfficientNet B2 query encoder · CIRR validation | Avg Recall@K: 63.77 -> 66.33 | Helped |
| Add global contrastive distillation to LAR (ablation) | ISA with EfficientNet B2 query encoder · CIRR validation | Avg Recall@K: 60.20 -> 66.33 | Helped |
| Replace LLM text regularization with LAR while retaining GCD (ablation) | ISA with EfficientNet B2 query encoder · CIRR validation | Avg Recall@K: 63.54 -> 66.33 | Helped |
| Add local alignment regularization to GCD (ablation) | ISA with EfficientNet B2 query encoder · CIRCO validation | Avg mAP@K: 11.43 -> 14.52 | Helped |
| Add global contrastive distillation to LAR (ablation) | ISA with EfficientNet B2 query encoder · CIRCO validation | Avg mAP@K: 8.16 -> 14.52 | Helped |
| Replace LLM text regularization with LAR while retaining GCD (ablation) | ISA with EfficientNet B2 query encoder · CIRCO validation | Avg mAP@K: 12.05 -> 14.52 | Helped |
| Add cross-attention to ATL (ablation) | ISA with EfficientNet B2 query encoder · CIRR validation | Avg Recall@K: 64.37 -> 66.33 | Helped |
| Add self-attention to ATL (ablation) | ISA with EfficientNet B2 query encoder · CIRR validation | Avg Recall@K: 65.08 -> 66.33 | Helped |
| Add cross-attention to ATL (ablation) | ISA with EfficientNet B2 query encoder · CIRCO validation | Avg mAP@K: 11.49 -> 14.52 | Helped |
| Add self-attention to ATL (ablation) | ISA with EfficientNet B2 query encoder · CIRCO validation | Avg mAP@K: 12.77 -> 14.52 | Helped |
Surprising findings
- The asymmetric setup with a lightweight query encoder outperforms the symmetric variant that uses the BLIP visual encoder on the query side.
- Training the BLIP visual encoder as query model with the gallery fixed reaches only 65.15 average on CIRR validation, below the lightweight-query model, attributed to reduced batch size for the heavy model.
Limitations
- Performance degrades with excessive token length, as some tokens attend to background or noisy patterns.
- Failure cases involve wrong associated concepts, rare words and abstract modifiers.
- Only evaluated with BLIP; CLIP lacks a matching encoder sharing the word embedding space, so the CLIP variant was not reported.
Experimental setup
Datasets: CC3M CIRR FashionIQ CIRCO ImageNet
Benchmarks: CIRR FashionIQ CIRCO
Metrics: Recall@K mAP@K
Baselines: Pic2Word SEARLE Image-only Text-only Image + Text Ours(Sym) CIRPLANT ARTEMIS CLIP4Cir
Builds on: Pic2Word SEARLE BLIP
Closest papers by meaning
- Language-only Training of Zero-shot Composed Image Retrieval (CVPR 2024 · similarity 0.91)
- Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR 2024 · similarity 0.90)
- Context-I2W: Mapping Images to Context-Dependent Words for Accurate Zero-Shot Composed Image Retrieval (AAAI 2024 · similarity 0.90)
- Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval (CVPR 2024 · similarity 0.90)
- Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.90)
- Generative Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.