Research map / Vision-language and multimodal / CLIP and zero-shot
WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WIKIDO, a 384K-pair Wikipedia image-text retrieval benchmark with human-verified in-domain and out-of-domain test sets, and benchmarks CLIP, BLIP and BLIP-2 zero-shot and after fine-tuning.
Problem
Existing cross-modal retrieval benchmarks (MSCOCO, Flickr30K) are near-saturated and their OOD evaluation is weak because the evaluation and fine-tuning datasets are visually and linguistically similar, so they do not robustly measure out-of-distribution generalization of VLMs.
Approach
Build a retrieval benchmark from Wikipedia Diversity Observatory topic labels, holding out topics (food, paintings, medicine) that are measurably distant in CLIP and BERT/ResNet embedding space (MMD) as OOD test sets, with LLaVA-enhanced captions and human verification of validation and test captions.
Key results
- BLIP-2 zero-shot reaches R@1 about 66% on the WIKIDO OOD set versus about 81% on MSCOCO and 95% on Flickr.
- Fine-tuning on 100K WIKIDO pairs improves OOD R@1 by at most about 5% versus about 12% on the ID set.
- Scaling ID training data from 100K to 354K gives only minimal OOD gains for BLIP (ViT-L), with image-to-text R@1 at 63.6 to 64.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on WIKIDO 100K split | VLMs · WIKIDO ID Test set | Image→Text R@1: All models show >8% improvement for R@1 on the ID set | Helped |
| fine-tune on WIKIDO 100K split | VLMs · WIKIDO OOD Test set | Image→Text R@1: ≈5% | Helped |
| evaluate on OOD rather than ID test data | BLIP and BLIP-2 · WIKIDO | R@K: 3-7% | Hurt |
| evaluate on OOD rather than ID test data | CLIP · WIKIDO | R@K: ≈1% gap | No clear effect |
| fine-tune on MSCOCO | BLIP-2 (ViT-L) · Flickr Test set | Image→Text R@1: 95.3 -> 97.1 | Helped |
| increase ID training data from 100K to 200K samples (ablation) | BLIP (ViT-L) · WIKIDO ID Test set | Image→Text R@1: 72.6 -> 74.1 | Helped |
| increase ID training data from 200K to 354K samples (ablation) | BLIP (ViT-L) · WIKIDO ID Test set | Image→Text R@1: 74.1 -> 76.2 | Helped |
| increase ID training data from 100K to 200K samples (ablation) | BLIP (ViT-L) · WIKIDO OOD Test set | Image→Text R@1: 63.6 -> 64.4 | Helped |
| increase ID training data from 200K to 354K samples (ablation) | BLIP (ViT-L) · WIKIDO OOD Test set | Image→Text R@1: 64.4 -> 64.3 | No clear effect |
| add ITM fine-tuning objective (ablation) | BLIP (ViT-L) · WIKIDO OOD Test set | Image→Text R@1: 59.6 -> 63.6 | Helped |
| add ITM fine-tuning objective (ablation) | BLIP-2 (ViT-L) · WIKIDO OOD Test set | Image→Text R@1: 61.2 -> 70.9 | Helped |
| add ITG fine-tuning objective to ITC+ITM (ablation) | BLIP-2 (ViT-L) · WIKIDO OOD Test set | Image→Text R@1: 70.9 -> 72.1 | Helped |
Surprising findings
- Fine-tuning on MSCOCO substantially improves Flickr performance and vice versa, indicating the two standard benchmarks overlap and are weak OOD tests.
- Scaling in-distribution training data barely closes the ID-OOD gap, suggesting more diverse data is needed.
Limitations
- Only image-text retrieval is evaluated; generation and contextual understanding are not covered.
- Only the topic axis of diversity is used, and data is English-only.
- Enhanced training captions from LLaVA were not manually verified.
Experimental setup
Datasets: WIKIDO MSCOCO Flickr30K Wikipedia Diversity Observatory Wikipedia dumps
Benchmarks: WIKIDO ID test WIKIDO OOD test MSCOCO 5K test Flickr30K 5K test
Metrics: Recall@1 Recall@5 Recall@10
Baselines: CLIP BLIP BLIP-2
Builds on: CLIP BLIP BLIP-2 WIT LLaVA
Closest papers by meaning
- Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval (AAAI 2025 · similarity 0.91)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.90)
- Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal Representations (CVPR 2025 · similarity 0.90)
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.89)
- CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.