Research map / Vision-language and multimodal / Human and embodied modeling

Do Composed Image Retrieval Benchmarks Require Multimodal Composition?

NeurIPS 2026 · Vision-language and multimodal › Human and embodied modeling · approach: Composed and text-based image retrieval · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Audits four composed image retrieval benchmarks for unimodal shortcuts using eleven generalist multimodal embedding models, then human-validates the shortcut-free residue to build the CIRCUS evaluation splits.

Problem

Composed image retrieval benchmarks are assumed to require combining reference image and text edit, but many queries may be solvable from a single modality or are ill-posed, so benchmark scores may overestimate multimodal composition.

Approach

Use a pool of generalist multimodal embedding models as probes: a query counts as a unimodal shortcut if any model retrieves the target from text-only or image-only input; remaining queries are human-validated for ambiguity and target correctness, yielding cleaner composition-requiring splits (CIRCUS).

Key results

• Unimodal shortcuts solve 83.6% of CIRR, 32.2% of FashionIQ, 38.7% of LaSCo and 74.5% of CIRCO queries across eleven retrievers.
• Only 1,689 of 4,741 audited shortcut-free queries (35.6%) are human-validated as well-formed.
• Qwen3-VL-8B Recall@10 on CIRR falls from 70.1 (full) to 17.2 (shortcut-free) and 24.1 (validated).

Surprising findings

• Individual retrievers underestimate shortcut prevalence; the strongest single model flags 64.4% of CIRR queries versus 83.6% when aggregated.
• Overly broad queries account for 60-85% of invalid instances, so much residual difficulty comes from under-specification rather than compositional reasoning.

Limitations

• Conclusions depend on the evaluated retriever pool and the Recall@10 cutoff.
• Human validation is only a stratified sample for FashionIQ and LaSCo, so residual-set estimates carry sampling uncertainty.
• CIRCO analysis uses the validation split with only 220 queries because the test split is not public.

Experimental setup

Datasets: CIRR FashionIQ LaSCo CIRCO

Benchmarks: CIRR FashionIQ LaSCo CIRCO

Metrics: Recall@10 nDCG MRR composition gap

Baselines: E5-Omni GME-Qwen2VL LamRA LamRA-Qwen2.5VL MM-Embed Qwen3-VL-2B Qwen3-VL-8B Rzen-Embed VLM2Vec-V2 Gemini Embedding 2 Voyage MM-3.5

Builds on: CIRR FashionIQ LaSCo CIRCO LamRA MM-Embed

Closest papers by meaning

• ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval (CVPR 2025 · similarity 0.92)
• CoLLM: A Large Language Model for Composed Image Retrieval (CVPR 2025 · similarity 0.91)
• Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.91)
• Generative Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.91)
• Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.90)
• Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR 2024 · similarity 0.90)