Research map / Vision-language and multimodal / Human and embodied modeling
Generative Zero-Shot Composed Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Augments zero-shot composed image retrieval with a self-supervised latent diffusion model that generates pseudo-target images from reference image plus delta caption, then fuses their text embeddings into the retrieval query.
Problem
Zero-shot composed image retrieval relies on pseudo-token textual inversion, which leaves a representation gap between composed language-space embeddings and target image embeddings and loses visual content of the reference image.
Approach
Fine-tune a latent diffusion model, conditioned on CLIP composed prompt embeddings (reference image pseudo token plus caption), to reconstruct images from image-caption pairs without triplets; at inference, generate pseudo-target images from the reference and delta caption and fuse their composed text embeddings with the original query, as an add-on to existing ZS-CIR methods.
Key results
- On CIRR test with ViT-L/14, SEARLE + CIG-XL turbo raises Recall@1 from 24.24 to 26.72.
- LinCIR + CIG with ViT-G/14 improves CIRR Recall@1 from 35.25 to 36.05 and CIRCO mAP@5 from 19.71 to 20.64 with CIG-XL.
- Generated-image CLIP score against composed text embeddings is 0.2810 for SEARLE + CIG versus 0.2555 for SEARLE + SD (target images 0.2762).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CIG to LinCIR | LinCIR with ViT-G/14 · CIRR test | Recall@1: 35.25 -> 36.05 | Helped |
| Add CIG-XL turbo to SEARLE | SEARLE with ViT-L/14 · CIRR test | Recall@1: 24.24 -> 26.72 | Helped |
| Add CIG-XL to LinCIR | LinCIR with ViT-G/14 · Fashion-IQ validation, Average | R@10: 45.11 -> 45.92 | Helped |
| Add CIG-XL to LinCIR | LinCIR with ViT-G/14 · CIRCO test | mAP@5: 19.71 -> 20.64 | Helped |
| Add CIG to LinCIR | LinCIR with ViT-G/14 · GeneCIS test, Avg | R@1: 13.66 -> 13.84 | Helped |
| Use text embedding fusion instead of image-level fusion (ablation) | SEARLE + CIG · CIRR validation | Recall@1: 24.32 -> 26.17 | Helped |
| Use pseudo-token fusion instead of image-level fusion (ablation) | SEARLE + CIG · CIRR validation | Recall@1: 24.32 -> 25.64 | Helped |
| Use text embedding fusion instead of pseudo-token fusion (ablation) | SEARLE + CIG · CIRCO validation | mAP@5: 10.57 -> 10.98 | Helped |
| Use text embedding fusion instead of image-level fusion (ablation) | SEARLE + CIG · CIRCO validation | mAP@5: 10.43 -> 10.98 | Helped |
Surprising findings
- Generated pseudo-target images score higher CLIP similarity to composed text embeddings than the actual ground-truth target images.
- Fusing at the image level (CLIP features of generated image) performs worse than fusing composed text embeddings, which is contrary to the intuition that image-space fusion would help.
Limitations
- Gains on Fashion-IQ are smaller, attributed to the difficulty of generating clothing images.
- Additional inference cost of 2 s per image with Stable Diffusion and SDXL, which is slower than baseline retrieval.
- GeneCIS captions of 1-2 words are too short for effective generation, so improvements are modest.
Experimental setup
Datasets: CC3M (filtered 595K subset from Li et al.) CIRR CIRCO Fashion-IQ GeneCIS
Benchmarks: CIRR CIRCO Fashion-IQ GeneCIS
Metrics: Recall@K RecallSubset@K mAP@k CLIP score
Baselines: Pic2Word SEARLE LinCIR CIReVL TIRG Image-only Text-only Image + Text
Builds on: Stable Diffusion SEARLE LinCIR Pic2Word CLIP
Closest papers by meaning
- ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval (CVPR 2025 · similarity 0.94)
- Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval (CVPR 2024 · similarity 0.93)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.92)
- Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.92)
- Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR 2024 · similarity 0.91)
- Language-only Training of Zero-shot Composed Image Retrieval (CVPR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.