Research map / Vision-language and multimodal / Human and embodied modeling
Context-I2W: Mapping Images to Context-Dependent Words for Accurate Zero-Shot Composed Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Context-I2W, a context-dependent image-to-pseudo-word mapping network (Intent View Selector plus Visual Target Extractor on frozen CLIP ViT-L/14) that improves zero-shot composed image retrieval across four tasks without triplet supervision.
Problem
Zero-shot composed image retrieval (ZS-CIR) needs a reference-image representation that adaptively attends to the parts of the image relevant to varied manipulation descriptions, but prior methods map the whole image to one pseudo-word regardless of the description.
Approach
Make the image-to-pseudo-word mapping context-dependent: the description rotates the image feature space (Intent View Selector via elementwise gating with a context embedding), and learnable queries with cross-attention extract description-relevant local targets, combined with a global feature and trained with a contrastive loss on image-caption pairs.
Key results
- Domain conversion on ImageNet: average R10 12.9 and R50 27.6 vs Pic2Word 10.1 and 23.2 (about 3.6% average gain).
- CIRR object composition: R1 25.6 vs SEARLE-XL 24.2 and Pic2Word 23.9.
- Fashion-IQ attribute manipulation: average R10 27.8 vs SEARLE-XL 25.7, and R50 48.9 vs 46.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Context-I2W instead of Pic2Word for domain conversion | ZS-CIR · ImageNet domain conversion average | R10: 10.1 -> 12.9 | Helped |
| Use Context-I2W instead of Pic2Word for object composition | ZS-CIR · COCO object composition | R1: 11.5 -> 13.5 | Helped |
| Use Context-I2W instead of SEARLE-XL for object/scene manipulation | ZS-CIR · CIRR | R1: 24.2 -> 25.6 | Helped |
| Use Context-I2W instead of SEARLE-XL for attribute manipulation | ZS-CIR · Fashion-IQ | R10: 25.7 -> 27.8 | Helped |
| Add context embedding (ablation) | Context-I2W · CIRR | R1: 21.4 -> 25.6 | Helped |
| Add subject replacement with [REPLACE] (ablation) | Context-I2W · CIRR | R1: 24.4 -> 25.6 | Helped |
| Add Intent View Selector (ablation) | Context-I2W · CIRR | R1: 24.0 -> 25.6 | Helped |
| Add Visual Target Extractor (ablation) | Context-I2W · CIRR | R1: 23.3 -> 25.6 | Helped |
| Use adaptive gating to combine local and global features (ablation) | Context-I2W · CIRR | R1: 24.5 -> 25.6 | Helped |
| Use Context-I2W mapping instead of typical cross-attention (ablation) | Context-I2W · CIRR | R1: 17.0 -> 25.6 | Helped |
| Replace only the first subject rather than all nouns (ablation) | Context-I2W · CIRR | R1: 24.7 -> 25.6 | Helped |
| Use average pooling instead of max pooling (ablation) | Context-I2W · CIRR | R1: 24.3 -> 25.6 | Helped |
Surprising findings
- Replacing the cross-attention mapping with a standard description-to-image cross-attention drops performance by about 10.3% on average, larger than removing either module.
Limitations
- Struggles with images containing many small objects, as the learned queries have difficulty capturing fine-grained patch-level details.
- Larger model (65.3M parameters) and longer training (24h) than Pic2Word's 3-layer MLP.
Experimental setup
Datasets: Conceptual Captions ImageNet COCO CIRR Fashion-IQ
Benchmarks: ImageNet COCO CIRR Fashion-IQ
Metrics: Recall@1 Recall@5 Recall@10 Recall@50
Baselines: Pic2Word SEARLE-XL Image-only Text-only Image+Text Combiner TIRG ARTEMIS CIRPLANT MAAF
Builds on: Pic2Word CLIP DETR SEARLE
Closest papers by meaning
- Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.93)
- Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.91)
- ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval (CVPR 2025 · similarity 0.90)
- Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval (ICLR 2024 · similarity 0.90)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.90)
- Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.