Research map / Vision-language and multimodal / CLIP and zero-shot
Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes CLIP with category-level, bag-level image-text pairs built from GPT-4-generated class descriptions (visual, habitat, geographic, taxonomy) on fine-grained datasets to improve zero-shot classification on unseen classes.
Problem
Zero-shot classification of VLMs such as CLIP is weak in fine-grained domains where aligned image-caption data is scarce and CLIP does not exploit fine-grained attributes at test time.
Approach
Pair images with LLM-generated class descriptions at the category level (stochastic image-text pairing within a class) and train CLIP with a category-aware contrastive loss, then evaluate using LLM descriptions of novel classes, improving zero-shot transfer rather than only adding attributes at test time.
Key results
- CLIPFT+A improves CUB zero-shot accuracy from 50.54% (CLIP) to 53.34% and Stanford Cars from 69.72% to 71.63%, with FGVC Aircraft 29.27% to 36.41% (ViT-B/32, Table 1).
- Against CLIP-A-self with ViT-B/16 (16 images per class), CLIPFT+A reaches 81.82% on EuroSAT vs 70.50% for CLIP-A-self and 64.05% for CLIP.
- On NeWT multi-task evaluation, the iNaturalist-trained model improves mean average precision from 60.25 to 61.90, a 4.1% relative error reduction over CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CLIP with LLM-generated attributes rather than only use them at test time | CLIP ViT-B/32 · CUB, zero-shot test classes | accuracy: 50.71 -> 53.34 | Helped |
| Fine-tune CLIP with LLM-generated attributes rather than only use them at test time | CLIP ViT-B/32 · Stanford Cars, zero-shot test classes | accuracy: 69.47 -> 71.63 | Helped |
| Fine-tune CLIP with LLM-generated attributes rather than only use them at test time | CLIP ViT-B/32 · FGVC Aircrafts, zero-shot test classes | accuracy: 30.35 -> 36.41 | Helped |
| Fine-tune CLIP with LLM-generated attributes rather than only use them at test time | CLIP ViT-B/32 · Flowers 102, zero-shot test classes | accuracy: 75.37 -> 77.05 | Helped |
| Fine-tune CLIP with LLM-generated attributes rather than only use them at test time | CLIP ViT-B/32 · Food 101, zero-shot test classes | accuracy: 90.08 -> 93.71 | Helped |
| Use LLM-generated attributes at test time without fine-tuning | CLIP ViT-B/32 · Stanford Cars, zero-shot test classes | accuracy: 69.72 -> 69.47 | Hurt |
| Use LLM-generated attributes at test time without fine-tuning | CLIP ViT-B/32 · CUB, zero-shot test classes | accuracy: 50.54 -> 50.71 | No clear effect |
| Add habitat descriptions to visual attributes (ablation) | CLIPFT + A · CUB, zero-shot test classes | accuracy: 53.34 -> 54.01 | Helped |
| Add taxonomy and habitat descriptions to visual attributes (ablation) | CLIPFT + A · CUB, zero-shot test classes | accuracy: 53.34 -> 54.23 | Helped |
| Use habitat descriptions only instead of visual descriptions (ablation) | CLIPFT + A · CUB, zero-shot test classes | accuracy: 53.34 -> 53.69 | Helped |
| Use taxonomy information only instead of visual descriptions (ablation) | CLIPFT + A · CUB, zero-shot test classes | accuracy: 53.34 -> 53.07 | Hurt |
| Fine-tune with LLM-generated attributes rather than only use them at test time | CLIP ViT-B/16 · EuroSAT, zero-shot test classes | accuracy: 71.51 -> 81.82 | Helped |
Surprising findings
- Simply appending LLM-generated attributes to class names at test time does not improve pre-trained CLIP on fine-grained datasets (e.g., Stanford Cars drops from 69.72% to 69.47%)
- Stochastically pairing images with all category texts works better than biasing sampling toward similar pairs or using CLIP-based visibility masking
Limitations
- Training texts are LLM-generated and not verified at scale; manual vetting does not scale to large datasets
- Method relies on fine-grained class-level data and LLM knowledge of category names
- Gains on coarser datasets are smaller and the comparison to prior work uses specific splits
Experimental setup
Datasets: CUB-200 Flowers 102 Stanford Cars FGVC Aircraft Food101 EuroSAT ImageNet Caltech101 DTD Oxford Pets SUN397 UCF101
Benchmarks: CUB-200 Stanford Cars FGVC Aircraft Flowers 102 Food101 EuroSAT ImageNet NeWT
Metrics: top-1 accuracy mean average precision
Baselines: CLIP CLIP + A CLIPFT CLIP-A-self CoOp CoCoOp
Builds on: CLIP CoOp CuPL CLIP-Adapter FixMatch
Closest papers by meaning
- Bridge the Modality and Capability Gaps in Vision-Language Model Selection (NeurIPS 2024 · similarity 0.90)
- Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification (ICLR 2024 · similarity 0.90)
- TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.89)
- Zero-Shot Robustification of Zero-Shot Models (ICLR 2024 · similarity 0.89)
- S3A: Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment (AAAI 2024 · similarity 0.89)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.