Research map / Vision-language and multimodal / CLIP and zero-shot
Visual Classification via Description from Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace category-name matching with GPT-3-generated descriptors | CLIP ViT-L/14 · ImageNet | accuracy: 75.00 vs 71.58 | Helped |
| Replace category-name matching with GPT-3-generated descriptors | CLIP ViT-L/14 · ImageNetV2 | accuracy: 69.3 vs 65.33 | Helped |
| Replace category-name matching with GPT-3-generated descriptors | CLIP ViT-L/14 · CUB | accuracy: 63.46 vs 63.08 | Helped |
| Replace category-name matching with GPT-3-generated descriptors | CLIP ViT-L/14 · EuroSAT | accuracy: 48.66 vs 41.48 | Helped |
| Replace category-name matching with GPT-3-generated descriptors | CLIP ViT-L/14 · Places365 | accuracy: 40.55 vs 39.00 | Helped |
| Replace category-name matching with GPT-3-generated descriptors | CLIP ViT-L/14 · Food101 | accuracy: 92.44 vs 91.79 | Helped |
| Replace category-name matching with GPT-3-generated descriptors | CLIP ViT-L/14 · Oxford Pets | accuracy: 92.23 vs 88.25 | Helped |
| Replace category-name matching with GPT-3-generated descriptors | CLIP ViT-L/14 · Describable Textures | accuracy: 54.36 vs 51.33 | Helped |
| Use descriptors to classify post-training concepts | CLIP with GPT-3 descriptors · Wordle and Ever Given, top ten detections | recall: 100% vs 0% (Wordle); 100% vs 10% (Ever Given) | Helped |
| Replace GPT-3 descriptors with Wiktionary descriptions (ablation) | ImageNet classification by description, ViT-L/14 · ImageNet | accuracy: 68.87 vs 75.00 | Hurt |
| Replace max aggregation with mean aggregation of descriptor similarities (ablation) | ImageNet classification by description, ViT-L/14 · ImageNet | accuracy: 71.14 -> 73.15 | Helped |
Closest papers by meaning
- The Neglected Tails in Vision-Language Models (CVPR 2024 · similarity 0.91)
- Does VLM Classification Benefit from LLM Description Semantics? (AAAI 2025 · similarity 0.91)
- Why are Visually-Grounded Language Models Bad at Image Classification? (NeurIPS 2024 · similarity 0.91)
- Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification (ICLR 2024 · similarity 0.90)
- Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · similarity 0.90)
- Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.