Research map / Vision-language and multimodal / CLIP and zero-shot
Learning Customized Visual Models With Retrieval-Augmented Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Customize CLIP with relevant retrieved image-text pairs | CLIP ViT-L/14 · ImageNet-1K zero-shot | accuracy: 75.3 -> 78.1 | Helped |
| Increase the retrieval pool from LAION-400M to Web-800M | REACT with CLIP ViT-L/14 and 6M retrieved pairs · ImageNet-1K zero-shot | accuracy: 77.6 -> 78.5 | Helped |
| Customize with retrieved image-text pairs | CLIP · ELEVATER ICinW 20 datasets, zero-shot | average score: 56.8 -> 60.6 | Helped |
| Customize with retrieved image-text pairs | CLIP-L/14 · Flickr30K zero-shot, text-to-image R@1 | R@1: 68.7 -> 76.5 | Helped |
| Customize RegionCLIP with retrieved image-text pairs | RegionCLIP with CLIP ResNet-50 · MSCOCO open-vocabulary detection, novel classes | AP50: 14.2 -> 20.6 (+6.4) | Helped |
| Customize MaskCLIP using locked-text tuning | MaskCLIP ViT-B/16 · COCO Stuff semantic segmentation with key smoothing and prompt denoising | mIoU: 14.6 -> 18.2 (+3.6) | Helped |
| Add gated image blocks to locked-text tuning (ablation) | REACT · ImageNet-1K zero-shot | accuracy: 66.9 -> 68.6 | Helped |
| Freeze the text encoder instead of freezing the image encoder and tuning text (ablation) | CLIP ViT-B/32 customization · ImageNet-1K zero-shot | accuracy: 63.7 -> 66.9 | Helped |
| Train gated blocks rather than directly tuning the pretrained weights (ablation) | CLIP ViT-B/32 customization · ImageNet-1K zero-shot | accuracy: 64.3 -> 68.6 | Helped |
| Increase retrieved-pair count from 1M to 6M (ablation) | REACT · ImageNet-1K zero-shot | accuracy: 64.8 -> 68.6 | Helped |
| Use text-to-text retrieval rather than text-to-image retrieval alone (ablation) | REACT · ImageNet-1K zero-shot | accuracy: 65.8 -> 68.6 | Helped |
| Replace paired-caption supervision with pseudo-label supervision (ablation) | REACT customization with T2I/T2T data · ImageNet-1K zero-shot | accuracy: 68.6 -> 66.2 | Hurt |
Closest papers by meaning
- Retrieval-Enhanced Contrastive Vision-Text Models (ICLR 2024 · similarity 0.90)
- Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.87)
- RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning (ICML 2025 · similarity 0.87)
- Context-I2W: Mapping Images to Context-Dependent Words for Accurate Zero-Shot Composed Image Retrieval (AAAI 2024 · similarity 0.86)
- What If We Recaption Billions of Web Images with LLaMA-3? (ICML 2025 · similarity 0.86)
- Retaining Knowledge and Enhancing Long-Text Representations in CLIP through Dual-Teacher Distillation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.