Research map / Vision-language and multimodal / Human and embodied modeling
Composed and text-based image retrieval
Retrieves images from text, sketch, or reference-plus-modification queries with vision-language encoders.
78 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Human and embodied modeling, keyword cluster “retrieval · scene · captioning”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Leveling off Several papers from 2025–26 still report on 1 tracked benchmark, but the best result has barely moved since 2024.
16 papers in 2023–24 and 43 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 3.0% to 6.4%.
Open questions the data points to
- Papers report CIReVL on CIRCO (mAP@5) anywhere from 18.6 to 26.8: comparisons against it depend on whose number you take.
What looks settled
- CIRR test (Recall@1): best up to 2024 51.96, best in 2025–26 45.8 from 3 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Natural Language Object Retrieval (CVPR 2016 · 588 citations)
- 2017 · Person Search With Natural Language Description (CVPR 2017 · 534 citations)
- 2019 · LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search (CVPR 2019 · 60 citations)
- 2020 · Image Search With Text Feedback by Visiolinguistic Attention Learning (CVPR 2020 · 193 citations)
- 2021 · Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback (CVPR 2021 · 244 citations)
- 2022 · Action-Aware Embedding Enhancement for Image-Text Retrieval (AAAI 2022 · 37 citations)
- 2023 · Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style (CVPR 2023 · 63 citations)
- 2024 · Context-I2W: Mapping Images to Context-Dependent Words for Accurate Zero-Shot Composed Image Retrieval (AAAI 2024 · 51 citations)
- 2025 · ENCODER: Entity Mining and Modification Relation Binding for Composed Image Retrieval (AAAI 2025 · 28 citations)
- 2026 · HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrieval (AAAI 2026 · 5 citations)
Earliest papers here
- Natural Language Object Retrieval (CVPR 2016 · 588 citations)
- Sketch Me That Shoe (CVPR 2016 · 443 citations)
- Person Search With Natural Language Description (CVPR 2017 · 534 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
CIRR test · Recall@1 (higher is better)
4 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 51.96 | Sentence-level Prompts Benefit Composed Image Retrieval (ICLR 2024 · 5 citations) |
What has been tried
55 papers here report 664 measured changes (a component added, removed or swapped, compared with a baseline): helped 76%, no clear effect 3%, hurt 17%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add context embedding | CIRR · R1 | 21.4 -> 25.6 | Helped | Context-I2W: Mapping Images to Context-Dependent Words for Accurate Zero-Shot Composed Image Retrieval (AAAI 2024 · 51 citations) |
| add LFF to reference image and modification text | FashionIQ (Avg) · R@10 | 50.08 -> 56.13 | Helped | ENCODER: Entity Mining and Modification Relation Binding for Composed Image Retrieval (AAAI 2025 · 28 citations) |
| Add text-to-text generalisation loss LTT | ShoeV2 · Acc.@5 | 41.8 -> 47.3 | Helped | You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval (CVPR 2024 · 21 citations) |
| set teacher and student temperatures to 0.01 instead of 0.05 | ROxf-M · mAP | 74.67 -> 72.55 | Hurt | Let All Be Whitened: Multi-Teacher Distillation for Efficient Visual Retrieval (AAAI 2024 · 15 citations) |
| Use two-stage reasoning instead of one-stage reasoning | CIRCO · mAP@5 | 23.87 -> 21.73 | Hurt | Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval (CVPR 2025 · 14 citations) |
| use similarity-based image grouping | CIRR validation · R@1 | 52.45 -> 55.66 | Helped | Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval (CVPR 2024 · 14 citations) |
| replace the text encoder with one trained on LAION | FashionIQ · Mean | 50.03 -> 50.53 | Helped | FashionERN: Enhance-and-Refine Network for Composed Fashion Image Retrieval (AAAI 2024 · 13 citations) |
| Use uncertainty regularization alone, without fine-grained InfoNCE | Shoes · R@50 | 76.46% -> 77.41% | Helped | Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularization (ICLR 2024 · 12 citations) |
| use BEV-space retrieval instead of image-space retrieval | nuScenes-Retrieval Easy · Scene retrieval R@1 | 0.4644 -> 0.8194 | Helped | BEV-TSR: Text-Scene Retrieval in BEV Space for Autonomous Driving (AAAI 2025 · 10 citations) |
| Use text embedding fusion instead of image-level fusion | CIRR validation · Recall@1 | 24.32 -> 26.17 | Helped | Generative Zero-Shot Composed Image Retrieval (CVPR 2025 · 10 citations) |
| add cropped source views rather than using the entire target image | CIRR · R1 | 23.5 -> 27.2 | Helped | Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrieval (CVPR 2025 · 9 citations) |
| Fine-tune the SigLIP text module with LoRA | NL-DIR test · Recall@10 | 61.18 -> 79.40 | Helped | Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmark (CVPR 2025 · 8 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| CIReVL | CIRCO | mAP@5 | 18.6 – 26.8 | 2 papers, e.g. CVPR 2025 and ICML 2024 |
What reviewers will expect
Datasets and baselines named in the 27 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIRR · 10 of 27 papers
- FashionIQ · 9 of 27 papers
- CIRCO · 6 of 27 papers
- GeneCIS · 3 of 27 papers
- MS COCO · 3 of 27 papers
- Shoes · 2 of 27 papers
- LAION · 2 of 27 papers
- LaSCo · 2 of 27 papers
Baselines
- SigLIP · 6 of 27 papers
- CIReVL · 5 of 27 papers
- LinCIR · 5 of 27 papers
- Pic2Word · 5 of 27 papers
- CompoDiff · 4 of 27 papers
- SEARLE · 4 of 27 papers
- CLIP4CIR · 3 of 27 papers
- BLIP4CIR · 3 of 27 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 59 papers; from OpenAlex and the papers' affiliations).
- Chinese Academy of Sciences · 4 papers
- University of Chinese Academy of Sciences · 3 papers
- Kuaishou Technology · 3 papers
- Tsinghua University · 3 papers
- The University of Adelaide · 2 papers
- Peking University · 2 papers
- University of Maryland, College Park · 2 papers
- Amazon · 2 papers
- Zhejiang University · 2 papers
- Harbin Institute of Technology · 2 papers
Use it
Download BibTeX of 37 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Scene graph generation (126 papers)
- Language-grounded scene text and structure (124 papers)
- Multimodal retrieval-augmented modeling (85 papers)
- Concept-guided image captioning (69 papers)
- Neural image captioning (72 papers)
Elsewhere in the topic
- Vision-language pretraining (182 papers)
- Medical vision-language models (130 papers)
- fMRI visual decoding (106 papers)
- Human-like visual cognition in VLMs (97 papers)
- Multimodal and affective interaction modeling (87 papers)
- Language-grounded visual concept learning (83 papers)
- Layout-aware document understanding (73 papers)
- Human perception modeling (71 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).