Research map / Vision-language and multimodal / Cross-modal learning
Data Roaming and Quality Assessment for Composed Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LaSCo, a ~10x larger composed image retrieval dataset built by rephrasing VQA2.0 complementary pairs into transition texts with GPT-3, plus a modality-redundancy analysis and a BLIP-based early-fusion baseline (CASE) with a reverse-query auxiliary objective.
Problem
Composed image retrieval (CoIR) datasets are small, domain-specific or suffer from modality redundancy and false negatives, limiting progress on queries that truly require both image and text.
Approach
Generate CoIR triplets automatically ("data roaming") from VQA2.0 complementary image pairs by rephrasing (question, counter-answer) into transition text with GPT-3; propose CASE, a BLIP-initialized early-fusion cross-attention shift encoder trained with a reverse-query auxiliary objective and Recall@K surrogate loss, and a CLIP-based metric for modality redundancy.
Key results
- On FashionIQ, CASE reaches average R@10 48.79 and R@50 70.68, beating LF-CLIP (35.39 / 59.03).
- On CIRR test, CASE pre-trained on LaSCo improves over train-only CASE (R@1 48.68 vs 48.00), and zero-shot CASE-LaSCo.Ca reaches R@1 35.40 without training on CIRR.
- On the LaSCo validation set, CASE reaches R@1 7.08 and R@50 50.25 versus LF-CLIP 4.01 and 32.08.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CASE instead of LF-CLIP | FashionIQ retrieval · FashionIQ validation, average | R@10: 35.39 -> 48.79 | Helped |
| Use CASE instead of LF-CLIP | CIRR retrieval · CIRR test, Train | Recall@1: 33.59 -> 48.00 | Helped |
| Pre-train CASE on LaSCo.Ca rather than LaSCo | CASE zero-shot CIRR retrieval · CIRR test, Zero Shot | Recall@1: 30.89 -> 35.40 | Helped |
| Use CASE rather than LF-CLIP | LaSCo retrieval · LaSCo validation | R@1: 4.01 -> 7.08 | Helped |
| Add the Reverse-Query objective (ablation) | CASE on FashionIQ · FashionIQ validation | R@1: 19.98 -> 20.69 | Helped |
| Add the Reverse-Query objective (ablation) | CASE on CIRR · CIRR validation | R@1: 47.50 -> 47.96 | No clear effect |
| Use Recall@K surrogate loss instead of contrastive loss (ablation) | CASE on FashionIQ · FashionIQ validation | R@50: 69.98 -> 70.68 | Helped |
| Use Recall@K surrogate loss instead of contrastive loss (ablation) | CASE on CIRR · CIRR validation | R@10: 89.07 -> 88.88 | Hurt |
| Fine-tune rather than freeze ViT parameters (ablation) | CASE on FashionIQ · FashionIQ validation | R@50: 69.85 -> 70.68 | Helped |
| Fine-tune rather than freeze ViT parameters (ablation) | CASE on FashionIQ · FashionIQ validation | R@1: 21.03 -> 20.69 | Hurt |
| Fine-tune rather than freeze ViT parameters (ablation) | CASE on CIRR · CIRR validation | R@1: 48.07 -> 47.96 | No clear effect |
| Vary CASE training batch size across 32–4096 (ablation) | CASE on FashionIQ · FashionIQ validation | R@1: 0.91; 18.1; 18.62; 19.4; 19.4; 20.46; 20.1; 20.1 | Mixed |
Surprising findings
- CASE text-only variant reaches 34.89 average R@10 on FashionIQ, close to the full LF-CLIP baseline, indicating strong modality redundancy in FashionIQ.
- On CIRR, the text-only CASE variant surpasses several full methods on most metrics, indicating the image is often redundant.
Limitations
- Ground-truth labels contain false negatives since each query has a single target image.
- Relative improvement from LaSCo pre-training on CIRR is small (about 0.5-1%), attributed to modality redundancy in CIRR.
- Transition texts are GPT-3 generated, with some logic or phrasing failures in quality checks (about 8% judged not well-phrased).
Experimental setup
Datasets: VQA2.0 CIRR FashionIQ COCO captions
Benchmarks: FashionIQ CIRR LaSCo
Metrics: Recall@K R_subset@K Average Recall positive rate MOS
Baselines: CIRPLANT TRACE VAL CoSMo FashionVLP LF-CLIP LF-BLIP TIRG ARTEMIS MAAF
Builds on: BLIP CLIP VQA2.0 GPT-3 Recall@K surrogate loss
Closest papers by meaning
- CoVR: Learning Composed Video Retrieval from Web Video Captions (AAAI 2024 · similarity 0.91)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.91)
- Do Composed Image Retrieval Benchmarks Require Multimodal Composition? (NeurIPS 2026 · similarity 0.90)
- ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval (CVPR 2025 · similarity 0.89)
- Composed Video Retrieval via Enriched Context and Discriminative Embeddings (CVPR 2024 · similarity 0.89)
- CoLLM: A Large Language Model for Composed Image Retrieval (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.