Research map / Vision-language and multimodal / CLIP and zero-shot
Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KEDs, a zero-shot composed image retrieval framework that enriches CLIP pseudo-word token generation with a retrieved image-caption database and adds a second stream aligning pseudo-tokens with textual concepts via pseudo-triplets mined from CC3M.
Problem
Zero-shot composed image retrieval (retrieving a target image from a reference image plus text modification) where prior pseudo-word methods use only global image features and miss fine-grained attributes such as color, object number and layout.
Approach
A Bi-modality Knowledge-guided Projection network retrieves top-K images and captions from a 0.5M CC3M database and fuses them via cross-attention to build pseudo-word tokens; a second stream trained on pseudo-triplets mined from image-caption pairs with a semantic registration loss aligns the tokens with textual subject concepts; the two streams are linearly combined at inference.
Key results
- On ImageNet-R domain conversion, KEDs improves over prior zero-shot methods by +7.9% Recall@10 and +12.2% Recall@50 on average.
- On CIRR test, KEDs reaches 26.4% Recall@1, a 2.2-point gain over Pic2word-style baselines' SEARLE-XL (24.2).
- On Fashion-IQ validation, KEDs averages 26.8 R10 and 47.9 R50, versus 25.7/46.3 for SEARLE-XL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KEDs instead of Pic2word | Zero-shot composed image retrieval · ImageNet-R domain conversion | R10 (Average): 10.1 -> 18.0; +7.9% | Helped |
| Use KEDs instead of Pic2word | Zero-shot composed image retrieval · COCO object composition | R10: 33.4 -> 34.9; +1.5% | Helped |
| Use KEDs instead of SEARLE-XL | Zero-shot composed image retrieval · Fashion-IQ validation | R10 (Average): 25.7 -> 26.8 | Helped |
| Use KEDs instead of SEARLE-XL | Zero-shot composed image retrieval · CIRR test | R1: 24.2 -> 26.4; 2.2% | Helped |
| Add top-K retrieved image features (ablation) | KEDs bi-modality knowledge-guided projection · CIRR validation | R10: 67.0 -> 69.2 | Helped |
| Add top-K retrieved caption features (ablation) | KEDs bi-modality knowledge-guided projection · CIRR validation | R10: 66.7 -> 69.2 | Helped |
| Add the shared linear block (ablation) | KEDs bi-modality knowledge-guided projection · CIRR validation | R10: 60.9 -> 69.2 | Helped |
| Add textual concept alignment branch ϕA (ablation) | KEDs · CIRR validation | R10: 66.9 -> 69.2 | Helped |
| Add context to textual concept alignment (ablation) | KEDs textual concept alignment branch · CIRR validation | R10: 67.8 -> 69.2 | Helped |
| Add complementary extra captions (ablation) | KEDs textual concept alignment branch · CIRR validation | R10: 68.5 -> 69.2 | Helped |
| Use CC3M database features instead of CIRR features (ablation) | KEDs database · CIRR validation | R10: 68.9 -> 69.2 | No clear effect |
Surprising findings
- Substituting CC3M database features with CIRR or Fashion-IQ features at inference hurts performance, indicating the database must be domain-compatible with training features.
Limitations
- Performance on the Fashion-IQ and CIRR benchmarks is evaluated on validation/test splits with gains that are modest relative to baselines.
- Database features from other domains are not compatible with the CC3M-trained projection, limiting plug-and-play use of domain-specific databases.
Experimental setup
Datasets: CC3M ImageNet-R COCO Fashion-IQ CIRR
Benchmarks: ImageNet-R COCO object composition Fashion-IQ CIRR
Metrics: Recall@1 Recall@5 Recall@10 Recall@50
Baselines: Pic2word SEARLE-XL Image-only Text-only Image+Text Combiner CIRPLANT ARTEMIS MAAF TIRG
Builds on: Pic2word SEARLE CLIP faiss
Closest papers by meaning
- Language-only Training of Zero-shot Composed Image Retrieval (CVPR 2024 · similarity 0.90)
- Context-I2W: Mapping Images to Context-Dependent Words for Accurate Zero-Shot Composed Image Retrieval (AAAI 2024 · similarity 0.89)
- Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval (ICLR 2024 · similarity 0.89)
- Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.89)
- Generative Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.88)
- Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.