Research map / Vision-language and multimodal / CLIP and zero-shot
Retrieval-Enhanced Contrastive Vision-Text Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RECO, which refines frozen CLIP/LiT embeddings with cross-modal knowledge retrieved from a large external image-text memory via a single-layer transformer fusion module trained on CC12M, improving zero-shot fine-grained recognition.
Problem
Contrastive vision-text models like CLIP struggle with fine-grained entities that are rare or absent from pre-training data; zero-shot performance on fine-grained recognition is weak.
Approach
Retrieve neighbours within the same modality (image-to-image, text-to-text) from a large image-text memory, then fuse the retrieved items of the opposite modality (captions for images, images for texts) into frozen embeddings with a light one-layer transformer fusion trained with contrastive losses that align refined and original embeddings.
Key results
- RECO on CLIP-B/32 improves zero-shot Cars from 57.2 to 68.1 (+10.9), CUB from 52.8 to 63.0 (+10.2) and Flowers by +5.8.
- On OVEN, CLIP-L/14 + RECO reaches 12.3 harmonic mean vs 5.3 for CLIP-L/14 zero-shot, outperforming fine-tuned PaLI-3B and CLIP2CLIP.
- Uni-modal search with cross-modal fusion gives +9.0% average relative improvement vs +1.7% for cross-modal search.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RECO retrieval and fusion | CLIP-B/32 · Cars | top-1 accuracy: 57.2 -> 68.1; +10.9 | Helped |
| add RECO retrieval and fusion | CLIP-B/32 · CUB | top-1 accuracy: 52.8 -> 63.0; +10.2 | Helped |
| add RECO retrieval and fusion | CLIP-B/32 · Flowers | top-1 accuracy: +5.8 | Helped |
| add RECO retrieval and fusion | CLIP-L/14 · OVEN unseen | top-1 accuracy: 4.9 -> 13.3 | Helped |
| add RECO retrieval and fusion | CLIP-B/32 · COCO T→I | recall@1: +3.4 | Helped |
| use uni-modal rather than cross-modal search (ablation) | Transformer cross-modal fusion · Cars, CUB, Flowers, ImageNet-1k, Places365 | average relative improvement over CLIP: +1.7 % -> +9.0 % | Helped |
| use cross-modal rather than uni-modal fusion (ablation) | Transformer fusion · Cars, CUB, Flowers, ImageNet-1k, Places365 | average relative improvement over CLIP: +9.0 % -> $-$ 0.4 %; $-$ 1.9 % | Mixed |
| replace learned Transformer fusion with mean fusion (ablation) | RECO · Cars, CUB, Flowers, ImageNet-1k, Places365 | average relative improvement over CLIP: $+$ 9.0 % -> $-$ 21.7 % | Hurt |
| train fusion for both image and text branches rather than text only (ablation) | RECO · CUB | top-1 accuracy: 59.3 -> 63.0 | Helped |
| train fusion for both image and text branches rather than image only (ablation) | RECO · COCO T→I | recall@1: 31.3 -> 33.6 | Mixed |
| replace RECO with a scratch-trained MLP layer (ablation) | CLIP-B/32 · CUB | top-1 accuracy | Hurt |
| use full WebLI memory at inference after training with 1% WebLI (ablation) | RECO · CUB | top-1 accuracy | Helped |
Surprising findings
- Cross-modal search (retrieving opposite-modality neighbours directly) gives much smaller gains, and uni-modal fusion can hurt performance.
- Training an extra MLP layer on CLIP with the same data yields no gain and even degrades performance, so gains are not due to extra training.
Limitations
- Requires access to a large, rich image-text memory; best results use a large private memory (WebLI), while public LAION-400M gives smaller gains.
- Retrieval increases inference time (about 25% overhead).
Experimental setup
Datasets: Conceptual Captions 12M WebLI LAION-400M Stanford Cars CUB-200-2011 Oxford Flowers ImageNet-1k Places365 Stanford Dogs OVEN Flickr30k MS COCO
Benchmarks: Stanford Cars CUB-200-2011 Oxford Flowers ImageNet-1k Places365 Stanford Dogs OVEN Flickr30k MS COCO
Metrics: top-1 accuracy recall@1 harmonic mean
Baselines: CLIP LiT K-LITE RA-CLIP Align PaLI-17B CLIP2CLIP PaLI-3B FLAVA SuS-X
Builds on: CLIP LiT InfoNCE K-LITE RA-CLIP
Closest papers by meaning
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.90)
- Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder (NeurIPS 2026 · similarity 0.89)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.89)
- AmorLIP: Efficient Language-Image Pretraining via Amortization (NeurIPS 2025 · similarity 0.89)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
- FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.