Research map / Vision-language and multimodal / Human and embodied modeling
INQUIRE: A Natural World Text-to-Image Retrieval Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces INQUIRE, an expert-level text-to-image retrieval benchmark with 250 queries exhaustively labeled over the 5M-image iNat24 dataset, evaluating CLIP-style embedding models and multimodal LLMs as rerankers.
Problem
Existing text-to-image retrieval benchmarks are small, saturated, and built from captions with one relevant image per query, so they do not test expert-level retrieval with many relevant images per query.
Approach
A retrieval benchmark of 250 expert-sourced natural-world queries with comprehensive per-query relevance labels over a five-million-image dataset, with a full-rank task and a top-100 reranking task; multimodal LLMs are used as zero-shot rerankers via Yes/No token logits.
Key results
- Best full-rank CLIP model (SigLIP SO400m-14@384) reaches mAP@50 of 34.2; reranking top-50 with GPT-4o raises mAP@50 to 39.6 and top-100 to 43.7.
- On INQUIRE-Rerank, GPT-4o achieves AP 59.6 versus 50.1 for the best embedding model (SigLIP SO400m-14) and 22.1 for random.
- Queries with scientific terminology are harder; GPT-4o gets AP 53.3 on lingo queries versus 60.4 on non-lingo queries.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLIP ViT-L-14 instead of CLIP ViT-B-16, keeping OpenAI training data | CLIP · INQUIRE-Fullrank test | mAP@50: 10.4 -> 14.4 | Helped |
| Train CLIP ViT-B-16 on DFN curated data instead of OpenAI data | CLIP ViT-B-16 · INQUIRE-Fullrank test | mAP@50: 10.4 -> 15.1 | Helped |
| Scale the initial CLIP model from ViT-B-16 to ViT-H-14@378, using DFN | CLIP (DFN) · INQUIRE-Fullrank test | mAP@50: 15.1 -> 33.3 | Helped |
| Use CLIP ViT-B-16 (BioCLIP) instead of WildCLIP | CLIP ViT-B-16 · INQUIRE-Fullrank test | mAP@50: 7.4 -> 5.0 | Hurt |
| Rerank the top 50 CLIP ViT-H-14 results with GPT-4o | CLIP ViT-H-14 initial ranking · INQUIRE-Fullrank test | mAP@50: 33.3 -> 39.6 | Helped |
| Rerank the top 50 CLIP ViT-H-14 results with VILA-40B | CLIP ViT-H-14 initial ranking · INQUIRE-Fullrank test | mAP@50: 33.3 -> 37.4 | Helped |
| Increase the initial retrieval set from 50 to 100 for GPT-4o reranking | GPT-4o reranking · INQUIRE-Fullrank test | mAP@50: 39.6 -> 43.7 | Helped |
| Increase the initial retrieval set from 50 to 100 for LLaVA-v1.6-7B reranking | LLaVA-v1.6-7B reranking · INQUIRE-Fullrank test | mAP@50: 33.3 -> 32.3 | Hurt |
| Use GPT-4o instead of GPT-4V for reranking | Multimodal reranking · INQUIRE-Rerank test | AP: 47.8 -> 59.6 | Helped |
| Use a larger embedding model, SigLIP SO400m-14 instead of CLIP ViT-B-32 | Embedding retrieval model · INQUIRE-Rerank test | AP: 30.2 -> 50.1 | Helped |
| Use GPT-4o rather than VILA-40B | Multimodal reranking · INQUIRE-Rerank test | AP: 52.8 -> 59.6 | Helped |
| Rerank rather than keep the initial CLIP ViT-H-14 ranking | GPT-4o · INQUIRE-Fullrank test | mAP@50: 33.3 -> 43.7 | Helped |
Surprising findings
- BioCLIP, trained on natural-world data, underperforms generalist DFN and SigLIP models on non-species queries.
- Smaller CLIP models such as ViT-B-32 perform only slightly better than random on the rerank task.
Limitations
- Benchmark images come from iNaturalist with possible species label errors inherited into evaluation.
- Natural-world domain only, which may limit transfer of insights to other visual domains.
- Geographic bias toward North America, Europe, and Australasia.
Experimental setup
Datasets: iNat24 iNat21
Benchmarks: INQUIRE-Fullrank INQUIRE-Rerank
Metrics: mAP@50 (AP@k) nDCG@50 MRR
Baselines: CLIP ViT-B-32 CLIP ViT-L-14 CLIP ViT-H-14 SigLIP SO400m-14 BioCLIP WildCLIP BLIP-2 InstructBLIP PaliGemma LLaVA-v1.5-13B LLaVA-v1.6 VILA
Builds on: CLIP iNaturalist 2021 BioCLIP WildCLIP
Closest papers by meaning
- ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge (NeurIPS 2026 · similarity 0.89)
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- CoLLM: A Large Language Model for Composed Image Retrieval (CVPR 2025 · similarity 0.88)
- Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.88)
- RMIR: A Benchmark Dataset for Reasoning-Intensive Multimodal Image Retrieval (CVPR 2026 · similarity 0.88)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.