Research map / Vision-language and multimodal / CLIP and zero-shot
Reproducible Vision-Language Models Meet Concepts Out of Pre-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LAION-Beyond, a benchmark of out-of-pre-training concepts for OpenCLIP, and proposes few-shot and zero-shot name learning (FSNL, ZSNL) that tune only the name token embeddings of unseen concepts.
Problem
CLIP-style models are evaluated for generalization almost only on concepts seen during pre-training; their zero-shot transfer to open-world concepts absent from pre-training (OOP) is unclear and appears to fail due to poor image-text alignment of OOP names.
Approach
Tune only the token embeddings of OOP concept names with frozen image and text encoders (name-tuning), using shuffled contexts from similar concepts for few-shot learning and bipartite-graph matching of image-cluster centers to pseudo-names (with novel-class discovery from IP classes) for zero-shot learning.
Key results
- Zero-shot OpenCLIP shows a large OOP deficit on LAION-Beyond (400M), e.g. average IP accuracy far above OOP despite comparable clustering accuracy (IP 68.15 vs OOP 63.13 normalized clustering accuracy).
- FSNL 4-shot achieves 77.74 average OOP accuracy vs 58.34 for CoOp and 68.69 for CLIP-Adapter, with harmonic mean 62.55 vs 45.12 and 54.86.
- In open-world vocabulary transfer, FSNL reaches 60.72 average accuracy, +7.52 over the second-best method (CLIP-Adapter 53.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply FSNL few-shot name learning with shuffled contexts from similar concepts | OpenCLIP · LAION-Beyond (400M), OOP-to-IP | OOP accuracy: 18.67 -> 77.74 | Helped |
| Apply FSNL few-shot name learning with shuffled contexts from similar concepts | OpenCLIP · LAION-Beyond (400M), OOP-to-IP | H-mean: 26.92 -> 62.55 | Helped |
| Apply FSNL few-shot name learning with shuffled contexts from similar concepts | OpenCLIP · LAION-Beyond (400M), open-world vocabulary transfer | accuracy: 53.2 -> 60.72 | Helped |
| Use zero-shot name learning with image-text bipartite matching | OpenCLIP · LAION-Beyond (400M), Animals | accuracy: 21.8 -> 25.7 | Helped |
| Use zero-shot name learning with image-text bipartite matching | OpenCLIP · LAION-Beyond (400M), Pokemon | accuracy: 19.6 -> 62.7 | Helped |
| Use zero-shot name learning with image-text bipartite matching | OpenCLIP · LAION-Beyond (5B), Animals | accuracy: 31.1 -> 45.6 | Helped |
| Add OOP name-embedding tuning (ablation) | FSNL · LAION-Beyond (400M), Animals | accuracy | Helped |
| Add context augmentation using similar concepts (ablation) | FSNL · LAION-Beyond (400M), Animals | accuracy | Helped |
| Use categorical-ancestor branching to select similar concepts for context augmentation (ablation) | FSNL · LAION-Beyond (400M), Animals | accuracy | No clear effect |
| Use k-means to select similar concepts for context augmentation (ablation) | FSNL · LAION-Beyond (400M), Animals | accuracy | No clear effect |
| Use nearest neighbors to select similar concepts for context augmentation (ablation) | FSNL · LAION-Beyond (400M), Animals | accuracy | No clear effect |
Surprising findings
- Linear probe without text-encoder assistance rivals FSNL in 4 domains, suggesting prompt-tuning and adapters fail to align text with image features for OOP concepts.
- Scaling pre-training data (LAION-400M to 5B) does not guarantee better image-text alignment for OOP concepts; OOP zero-shot performance does not reliably improve with scale.
Limitations
- Zero-shot ZSNL depends on quality of clusters and novel-class discovery from IP classes; accuracy varies strongly across domains (e.g. Folk Art drops).
- OOP concept status is determined by LAION vocabulary matching, which may miss concepts or be imperfect.
- Linear probe and TaskRes cannot transfer to open vocabulary, so comparisons are limited.
Experimental setup
Datasets: LAION-400M LAION-2B LAION-5B LAION-Beyond
Benchmarks: LAION-Beyond (400M) LAION-Beyond (2B) LAION-Beyond (5B)
Metrics: top-1 accuracy class-balanced accuracy harmonic mean of OOP and IP accuracy normalized clustering accuracy
Baselines: OpenCLIP CoOp CoCoOp CLIP-Adapter TaskRes Linear Probe Learning-to-Name TransCLIP
Builds on: CLIP OpenCLIP CoOp CoCoOp CLIP-Adapter TransCLIP
Closest papers by meaning
- Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization (ICLR 2024 · similarity 0.90)
- Anchor-based Robust Finetuning of Vision-Language Models (CVPR 2024 · similarity 0.89)
- The Neglected Tails in Vision-Language Models (CVPR 2024 · similarity 0.88)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.88)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.88)
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.