Research map / Vision-language and multimodal / CLIP and zero-shot
Does VLM Classification Benefit from LLM Description Semantics?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a classname-free evaluation protocol and a training-free method that selects distinctive LLM-generated descriptions from the CLIP neighborhood of top-k candidate classes, to test whether description semantics (not ensembling noise) improve VLM classification.
Problem
Performance gains from LLM-generated class descriptions in CLIP zero-shot classification may come from a noise ensembling/test-time augmentation effect rather than genuine semantics, so it is unclear whether description semantics actually help.
Approach
Remove the class name from all descriptions except one (classname-free setup) and select descriptions training-free using a class-wise lookup similarity matrix over the top-k CLIP candidate neighborhood, keeping only descriptions that activate for one candidate more than for the other candidates.
Key results
- In the classname-free setup, the selected descriptions (5 per candidate) outperform DCLIP LLM assignments on most datasets, e.g. EuroSAT 49.89 vs 43.31 and DTD 47.23 vs 43.09 with ViT-B/32.
- With w_cls=0 (no classname), selected descriptions reach e.g. 50.16 on ImageNet and 41.53 on CUB200, while LLM and random assignments perform far worse.
- In the classname-containing setup, the method surpasses WaffleCLIP and FuDD on CUB200, EuroSAT, Places365, DTD and Flowers102 for some configurations, e.g. EuroSAT 58.57 vs 45.18 for FuDD with contrastive pool and 20 descriptions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select distinctive classname-free descriptions using VLM embedding-space feedback instead of LLM class assignments | CLIP ViT-B/32 · EuroSAT (classname-free) | accuracy: 43.31 -> 49.89 | Helped |
| Select distinctive classname-free descriptions using VLM embedding-space feedback instead of LLM class assignments | CLIP ViT-B/32 · CUB200 (classname-free) | accuracy: 51.79 -> 53.80 | Helped |
| Select distinctive descriptions rather than LLM-assigned descriptions | CLIP ViT-B/32 · ImageNetV2 (classname-free) | accuracy: 55.09 -> 55.48 | Helped |
| Select distinctive descriptions rather than LLM-assigned descriptions | CLIP ViT-B/32 · DTD (classname-free) | accuracy: 43.09 -> 47.23 | Helped |
| Use local top-k evaluation for LLM assignments instead of global evaluation (ablation) | LLM descriptions · ImageNet (classname-free) | accuracy: 61.99 -> 61.99 | No clear effect |
| Use random description assignments instead of LLM assignments (ablation) | CLIP ViT-B/32 · CUB200 (classname-free) | accuracy: 51.79 -> 52.00 | Helped |
| Set classname weight to zero, relying on selected classname-free descriptions (ablation) | Ours · EuroSAT (classname-free) | accuracy: 28.11 -> 43.24 | Helped |
| Replace LLM assignments with selected descriptions and use maxing instead of ensembling (ablation) | CLIP ViT-B/32 · EuroSAT | accuracy: 43.29 -> 53.10 | Helped |
| Replace LLM assignments with selected descriptions and use maxing instead of ensembling (ablation) | CLIP ViT-B/32 · ImageNet | accuracy: 55.82 -> 54.42 | Hurt |
| Use maxing rather than ensembling for LLM-assigned descriptions (ablation) | CLIP ViT-B/32 · Places365 | accuracy: 40.47 -> 37.21 | Hurt |
| Increase the number of preliminary candidate labels k (ablation) | Ours with DCLIP pool · CUB200 (classname-included) | accuracy: 55.90 -> 53.30 | Hurt |
| Increase the number of preliminary candidate labels k (ablation) | Ours with DCLIP pool · Flowers102 (classname-included) | accuracy: 66.11 -> 70.14 | Helped |
Surprising findings
- LLM-assigned descriptions perform similarly to random assignments in the classname-free setting, suggesting they add little distinctive semantics.
- Maxing (ensembling-free) evaluation of LLM descriptions falls below the ensembling baseline in 6 of 7 datasets, while selected descriptions exceed it in 5 of 7.
Limitations
- Description pool is fixed and generated by LLMs; the method does not refine the pool with selection feedback.
- Gains on ImageNet and ImageNetV2 are small or mixed, attributed to noisy backgrounds and less sensitivity to descriptions.
- Performance depends on hyperparameters k, n and w_cls, with different optimal values per dataset.
Experimental setup
Datasets: ImageNet ImageNetV2 CUB200-2011 EuroSAT Places365 DTD Flowers102 DCLIP description pool contrastive prompting description pool
Benchmarks: ImageNet ImageNetV2 CUB200 EuroSAT Places365 DTD Flowers102 Food101 Oxford Pets
Metrics: top-1 accuracy
Baselines: CLIP DCLIP WaffleCLIP FuDD LaBo
Builds on: DCLIP CuPL FuDD WaffleCLIP CLIP
Closest papers by meaning
- Why are Visually-Grounded Language Models Bad at Image Classification? (NeurIPS 2024 · similarity 0.90)
- Large Language Models are Good Prompt Learners for Low-Shot Image Classification (CVPR 2024 · similarity 0.89)
- Bridge the Modality and Capability Gaps in Vision-Language Model Selection (NeurIPS 2024 · similarity 0.89)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.88)
- VladVA: Discriminative Fine-tuning of LVLMs (CVPR 2025 · similarity 0.88)
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.