Research map / Vision-language and multimodal / Human and embodied modeling
CoLLM: A Large Language Model for Composed Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoLLM, an LLM-based composed image retrieval model trained on triplets synthesized on-the-fly from image-caption pairs (Slerp embedding interpolation and text templates), plus the MTCIR synthetic dataset (3.4M pairs) and refined CIRR/Fashion-IQ validation sets.
Problem
Composed image retrieval lacks large-scale labeled triplets (reference image, modification text, target image), and existing zero-shot or synthetic approaches use shallow fusion, unnatural text, or noisy/ambiguous benchmarks.
Approach
Synthesize CIR triplets on-the-fly from image-caption pairs by Slerp-interpolating the target image embedding with its in-batch nearest neighbor and combining captions via templates, then use an LLM (with image adapter) to produce composed query embeddings trained with contrastive loss.
Key results
- Zero-shot CoLLM with CLIP-L/14 reaches 20.3 mAP@5 on CIRCO, state-of-the-art among compared methods without triplet training.
- Fine-tuning CoLLM (BLIP-L) on MTCIR gives CIRR R@1 of 45.8 on the test set vs 40.1 for Omkar et al. (trained on WebCoVR).
- CoLLM on the refined CIRR validation set with MTCIR achieves R@1 of 60.4, outperforming BLIP-L (MTCIR) at 53.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on MTCIR instead of WebCoVR | CoLLM with BLIP-L/16 · CIRR test | Recall@1: 40.1 -> 45.8 | Helped |
| Fine-tune on MTCIR instead of LaSCo | CoLLM with BLIP-L/16 · CIRR test | Recall@1: 43.2 -> 45.8 | Helped |
| Add reference image embedding interpolation (ablation) | CoLLM pre-training · CIRCO | mAP sum: 14.7 -> 52.8 | Helped |
| Add reference image embedding interpolation (ablation) | CoLLM pre-training · CIRR | Recall sum: 170.1 -> 194.5 | Helped |
| Add modification text interpolation (ablation) | CoLLM pre-training · CIRCO | mAP sum: 14.5 -> 52.8 | Helped |
| Add modification text interpolation (ablation) | CoLLM pre-training · CIRR | Recall sum: 176.4 -> 194.5 | Helped |
| Include unimodal image-only and text-only queries during training (ablation) | CoLLM pre-training · CIRCO | mAP sum: 14.5 -> 52.8 | Helped |
| Include unimodal image-only and text-only queries during training (ablation) | CoLLM pre-training · CIRR | Recall sum: 99.7 -> 194.5 | Helped |
| Use the nearest in-batch neighbor instead of a random sample (ablation) | CoLLM reference image embedding interpolation · CIRCO | mAP sum: 46.7 -> 52.8 | Helped |
| Use the nearest in-batch neighbor instead of a random sample (ablation) | CoLLM reference image embedding interpolation · CIRR | Recall sum: 182.2 -> 194.5 | Helped |
| Use an LLEM fine-tuned for text retrieval instead of Mistral-7B (ablation) | CoLLM · CIRCO | mAP@5: 19.8 -> 19.6 | Helped |
Surprising findings
- Fine-tuning CoLLM on LaSCo led to notable performance degradation, while MTCIR improved BLIP-L across the board.
- Using the nearest in-batch neighbor for reference embedding interpolation outperforms a random in-batch sample (CIRCO 52.8 vs 46.7 mAP).
Limitations
- Only LLMs/LLEMs were explored; pre-trained MLLMs were not used.
- Triplet synthesis generates a single visual token, limiting detailed image information.
- Refined benchmarks still contain noise from original image pairs.
Experimental setup
Datasets: CC3M LAION LLaVA-558K MTCIR CIRR Fashion-IQ CIRCO LaSCo WebCoVR
Benchmarks: CIRCO CIRR Fashion-IQ refined CIRR refined Fashion-IQ
Metrics: recall@k mAP
Baselines: PALAVRA SEARLE Slerp-TAT CIReVL Pic2World LinCIR ContextI2W MCL CompoDiff MagicLens CoVR2 CASE
Builds on: SlerpTAT CIReVL MCL LLM2Vec-style LLM embeddings CLIP
Closest papers by meaning
- Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR 2024 · similarity 0.92)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.92)
- Do Composed Image Retrieval Benchmarks Require Multimodal Composition? (NeurIPS 2026 · similarity 0.91)
- ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval (CVPR 2025 · similarity 0.90)
- Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.89)
- Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.