Research map / Vision-language and multimodal / CLIP and zero-shot
Understanding Retrieval-Augmented Task Adaptation for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic study plus theory of retrieval-augmented task adaptation for CLIP in low-data regimes, showing image-to-image retrieval beats text-to-image retrieval and that logit ensembling with the zero-shot model is critical.
Problem
Understanding how retrieval from web-scale databases (LAION-5B) impacts adaptation of contrastive vision-language models like CLIP in low-data regimes, which remained an open question.
Approach
Disentangles retrieval-augmented CLIP adaptation into retrieval method (uni-modal I2I vs cross-modal T2I) and logit ensembling with zero-shot predictions, and provides theorems on modality gap, retrieval distribution shift, and ensemble benefit.
Key results
- I2I retrieval consistently outperforms T2I retrieval across shots and datasets; at 12 shots the gap between I2I and ID-retrieval oracle can be as small as 1% on average.
- Without logit ensemble with zero-shot CLIP, retrieval-only adaptation degrades substantially; on Textures (K=16, RN50) ensemble improves over RET by about 9 points (55.32 vs 45.86 with 8 seeds).
- On Textures the average classwise improvement is 12.5% for Ensemble over ZOCLIP versus 3.1% for RET alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use I2I retrieval instead of T2I retrieval | CLIP retrieval-augmented adaptation · downstream datasets across retrieval shots | accuracy | Helped |
| add I2I retrieved samples to zero-shot CLIP | CLIP retrieval-augmented adaptation · downstream datasets | accuracy: 1% on average (12 shots) | Helped |
| use T2I retrieval instead of zero-shot CLIP | CLIP retrieval-augmented adaptation · multiple downstream datasets | accuracy | No clear effect |
| ensemble zero-shot logits with I2I retrieval logits | CLIP retrieval-based adaptation · downstream datasets | accuracy | Helped |
| use knowledge-augmented prompts for T2I retrieval instead of original prompts (ablation) | CLIP RN50 T2I retrieval · 7-dataset average, 8-shot | accuracy: 67.77 -> 68.86 | Helped |
| use I2I retrieval instead of knowledge-augmented T2I retrieval | CLIP RN50 retrieval-augmented adaptation · 7-dataset average, 8-shot | accuracy: 68.86 -> 72.91 | Helped |
| increase I2I retrieval seed images from 2 to 8 per class (ablation) | RET · Textures, K=16 | accuracy: 38.48 -> 45.86 | Helped |
| increase I2I retrieval seed images from 2 to 8 per class (ablation) | Ensemble · Textures, K=16 | accuracy: 51.77 -> 55.32 | Helped |
| increase I2I retrieval seed images from 2 to 8 per class (ablation) | Ensemble (F) · Textures, K=16 | accuracy: 57.98 -> 62.94 | Helped |
| increase I2I retrieval seed images from 2 to 8 per class (ablation) | ZOCLIP · Textures, K=16 | accuracy: 42.79 -> 42.79 | No clear effect |
| combine retrieved samples with few-shot ID samples (ablation) | CLIP I2I retrieval adaptation · 7-dataset average, K=16 | accuracy | Helped |
| use a larger CLIP backbone (ablation) | CLIP retrieval-augmented adaptation · average across datasets | accuracy | Helped |
Surprising findings
- Knowledge-augmented LLM-generated prompts for T2I retrieval give only modest gains compared with I2I retrieval.
- T2I retrieval can retrieve images containing the class name as rendered text rather than the object, introducing misaligned inductive biases.
Limitations
- Theoretical results rely on realistic but strong assumptions (sample representativeness, cluster-structured retrieved data) and bounds are loose in places.
- Experiments are limited to seven classification datasets and CLIP backbones; results depend on LAION-5B availability.
Experimental setup
Datasets: LAION-5B Caltech101 Birds200 Food101 OxfordPets Flowers102 Textures UCF101
Benchmarks: Caltech101 Birds200 Food101 OxfordPets Flowers102 Textures UCF101
Metrics: accuracy 0-1 risk
Baselines: Zero-shot CLIP Tip-Adapter-F T2I retrieval ID retrieval oracle
Builds on: CLIP Tip-Adapter LAION-5B clip-retrieval
Closest papers by meaning
- Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective (NeurIPS 2024 · similarity 0.88)
- Vision-Language Models Create Cross-Modal Task Representations (ICML 2025 · similarity 0.88)
- Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval (AAAI 2025 · similarity 0.88)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.