Research map / Vision-language and multimodal / CLIP and zero-shot
The Neglected Tails in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Estimates concept frequency in VLM pretraining texts (LAION) using LLM-derived synonyms and LLM filtering, shows long-tailed concept distributions drive zero-shot VLM failures, and proposes REAL (REAL-Prompt and REAL-Linear) using frequent synonyms and text-retrieved balanced pretraining data.
Problem
Vision-language models have highly imbalanced zero-shot accuracy across visual concepts, and the cause (concept frequency in web-scale pretraining data) has not been measured directly.
Approach
Count concept frequency in pretraining captions via LLM-enumerated synonyms, string matching, and LLM filtering; then replace class names with their most frequent synonyms in prompts (REAL-Prompt) and train a linear classifier on a class-balanced set of text-retrieved pretraining images (REAL-Linear), avoiding costly VLM-based feature retrieval.
Key results
- REAL-Prompt averages 66.0 across eight zero-shot datasets vs 62.6 for OpenAI templates, outperforming DCLIP and CuPL prompts.
- REAL-Linear reaches 67.8 average, about 3% above REACT (10K per concept) while using 500 images per concept, 400x less storage and 10,000x less training time.
- REAL-Prompt improves Flowers accuracy from 68.0 to 76.6 and EuroSAT from 51.5 to 57.5 over OpenAI templates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace original class names with their most frequent pretraining synonyms | REAL-Prompt · Average of eight standard datasets | mean per-class accuracy: 62.6 -> 66.0 | Helped |
| retrieve 500 balanced pretraining examples per concept and train a linear classifier | REAL-Linear · Average of eight standard datasets | mean per-class accuracy: 65.5 -> 67.8 | Helped |
| use synonyms as well as original concept names for retrieval, rather than original names alone (ablation) | REAL-Linear · standard zero-shot recognition datasets | mean per-class accuracy: +4% | Helped |
| train the linear classifier with both text and image features using cross-modal adaptation, rather than image features only (ablation) | REAL-Linear · standard zero-shot recognition datasets | mean per-class accuracy: 6.4% | Helped |
| increase retrieval size from 100 to 500 examples per concept (ablation) | REAL-Linear · standard zero-shot recognition datasets | mean per-class accuracy: 0.9% | Helped |
| add REAL-Prompt synonym replacement to OpenAI prompt templates (ablation) | OpenAI templates · Flowers | mean per-class accuracy: 68.0 -> 76.6 (+8.6) | Helped |
| add REAL-Prompt synonym replacement to CuPL prompts (ablation) | CuPL · Flowers | mean per-class accuracy: 65.8 -> 72.3 (+6.5) | Helped |
| add REAL-Prompt synonym replacement to DCLIP prompts (ablation) | DCLIP · Food | mean per-class accuracy: 80.1 -> 80.0 (-0.1) | No clear effect |
| add REAL-Linear to REACT Locked-Text (ablation) | REACT Locked-Text · ImageNet | mean per-class accuracy: 65.7 -> 67.7 (+2.0) | Helped |
| add REAL-Linear to REACT Gated-Image (ablation) | REACT Gated-Image · ImageNet | mean per-class accuracy: 64.2 -> 66.9 (+2.7) | Helped |
| use frequent synonyms in prompts instead of OpenAI templates | REAL-Prompt · ImageNet tail classes | mean per-class accuracy: 55.2 -> 56.2+1.0 | Helped |
| retrieve balanced pretraining data and train a linear classifier instead of using OpenAI templates | REAL-Linear · ImageNet tail classes | mean per-class accuracy: 55.2 -> 58.9+3.7 | Helped |
Surprising findings
- Balanced sampling by query in MetaCLIP still yields a long-tailed concept distribution, with over half of queries having fewer than 50 samples.
- DALL-E 3 and SD-XL fail to generate rare concepts such as bank swallow and thorn apple, but succeed when prompted with frequent synonyms.
Limitations
- Concept frequency precision and recall cannot be accurately evaluated without ground-truth pretraining labels.
- Text-only estimation may miss visual concepts present in images but absent from captions.
- LLM-based filtering of each caption-concept pair is time-consuming.
Experimental setup
Datasets: LAION-400M LAION-2B ImageNet Flowers Cars Aircraft Pets Food DTD EuroSAT CUB
Benchmarks: ImageNet Flowers Cars Aircraft Pets Food DTD EuroSAT CUB ImageNet-V2 ImageNet-Adversarial ImageNet-Rendition
Metrics: mean per-class accuracy head vs tail accuracy
Baselines: CLIP OpenAI templates DCLIP CuPL REACT (Locked-Text) REACT (Gated-Image) NeuralPriming
Builds on: CLIP OpenCLIP REACT WiSE-FT CuPL
Closest papers by meaning
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.92)
- Bridge the Modality and Capability Gaps in Vision-Language Model Selection (NeurIPS 2024 · similarity 0.90)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.90)
- From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibration (CVPR 2025 · similarity 0.90)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.89)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.