Research map / Image generation / Text-to-image diffusion
Imagine and Seek: Improving Composed Image Retrieval with an Imagined Proxy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free plug-in method for zero-shot composed image retrieval that generates an imagined proxy image (LLM layout + MIGC++ generation) and fuses it with query and text features via a balanced similarity metric.
Problem
Zero-shot composed image retrieval (ZS-CIR) relies on projecting the query image into text space, losing visual detail and failing to align fine-grained attributes in complex relative captions.
Approach
Use an LLM to infer a proxy image layout from BLIP-2 captions and the relative caption, generate a proxy image with controllable generation (MIGC++ with query-image conditioning), then combine proxy, query and semantic-perturbation features into a robust proxy feature and multiply text and proxy similarities in a balance metric for retrieval.
Key results
- On CIRR with LDRE backbone (ViT-G/14), IP-CIR reaches Recall@10 of 80.00 vs 77.25 for LDRE alone, and Recall@1 of 39.25 vs 36.15.
- On CIRCO with LDRE (ViT-G/14), mAP@10 rises from 32.24 to 34.26.
- On FashionIQ with LinCIR (ViT-G/14), average Recall@10 improves from 45.11 to 45.74.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add IP-CIR to LDRE | LDRE with ViT-G/14 · CIRR test | Recall@10: 77.25 -> 80.00 | Helped |
| Add IP-CIR to LDRE | LDRE with ViT-L/14 · CIRR test | Recall@10: 67.54 -> 71.21 | Helped |
| Add IP-CIR to LDRE | LDRE with ViT-G/14 · CIRCO test | mAP@10: 32.24 -> 34.26 | Helped |
| Add IP-CIR to LDRE | LDRE with ViT-L/14 · CIRCO test | mAP@10: 24.03 -> 27.41 | Helped |
| Add IP-CIR to LinCIR | LinCIR with ViT-G/14 · FashionIQ val, average | Recall@10: 45.11 -> 45.74 | Helped |
| Add proxy-image features and weighted-sum similarity (ablation) | LDRE-G · CIRR test | Recall@10: 77.25 -> 77.52 | Helped |
| Add proxy-image features and weighted-sum similarity (ablation) | LDRE-G · CIRCO test | mAP@10: 32.24 -> 32.41 | Helped |
| Add robust proxy features to PI (ablation) | LDRE-G + PI · CIRR test | Recall@10: 77.52 -> 79.47 | Helped |
| Add robust proxy features to PI (ablation) | LDRE-G + PI · CIRCO test | mAP@10: 32.41 -> 33.59 | Helped |
| Add the balance metric (ablation) | LDRE-G + PI + RP · CIRR test | Recall@10: 79.47 -> 80.00 | Helped |
| Add the balance metric (ablation) | LDRE-G + PI + RP · CIRCO test | mAP@10: 33.59 -> 34.26 | Helped |
| Generate proxy images with the proposed layout-to-image approach rather than SD (ablation) | LDRE-G proxy-image generation · CIRCO test | mAP@10: 33.50 -> 34.26 | Helped |
Surprising findings
- Directly using proxy image similarity without robust-proxy composition reduces recall at k=1 and k=50 on CIRR, indicating the imagined proxies introduce noise.
Limitations
- Proxy images can contain noise that distracts retrieval, lowering some recall metrics when used alone.
- Gains on FashionIQ are small and Toptee subset drops in Recall@10.
- Depends on LLM layout quality and a large 32B LLM plus image generator per query, adding inference cost.
Experimental setup
Datasets: CIRR CIRCO FashionIQ
Benchmarks: CIRR CIRCO FashionIQ
Metrics: Recall@K RecallSubset@K mAP@K
Baselines: LinCIR LDRE CIReVL Pic2Word SEARLE
Builds on: LDRE LinCIR MIGC++ MIGC ELITE
Closest papers by meaning
- ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval (CVPR 2025 · similarity 0.89)
- Language-only Training of Zero-shot Composed Image Retrieval (CVPR 2024 · similarity 0.88)
- Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.88)
- Generative Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.88)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.88)
- Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.