Research map / Vision-language and multimodal / CLIP and zero-shot
Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FuDD, a zero-shot method that identifies ambiguous classes per image with CLIP and uses an LLM (GPT-3.5) to generate pairwise differential class descriptions for a follow-up classification step.
Problem
Zero-shot VLM image classification with LLM-generated class attribute descriptions ignores which other classes in the dataset are confusable, so selected attributes may not help distinguish the target classes.
Approach
Detect the k most similar (ambiguous) classes per image using the VLM's initial prediction, generate pairwise visually differentiating descriptions between them with an LLM via few-shot prompting, and re-classify using only those ambiguous classes and differential descriptions.
Key results
- FuDD outperforms naive LLM-generated descriptions by 1.44% (ViT-B/32) and 2.41% (ViT-L/14@336px) on average across 12 datasets, with up to 13.95 points on EuroSAT.
- Most of the gains come from describing the top ~5 most ambiguous classes; k=10 achieves similar improvements to k=|C|.
- Fine-tuned Llama 2 7B makes 48% of pairwise descriptions useful for differentiation on EuroSAT vs 19% before fine-tuning, with up to 23.41 points accuracy gain on EuroSAT with ViT-L/14.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FuDD differential descriptions instead of naive LLM-generated descriptions | CLIP ViT-L/14@336px · EuroSAT | accuracy: ↑13.95 | Helped |
| Use FuDD differential descriptions instead of naive LLM-generated descriptions | CLIP ViT-L/14@336px · CUB | accuracy: ↑0.88 | Helped |
| Use FuDD differential descriptions instead of naive LLM-generated descriptions | CLIP ViT-L/14@336px · DTD | accuracy: ↑1.86 | Helped |
| Use FuDD differential descriptions instead of naive LLM-generated descriptions | CLIP ViT-L/14@336px · Food101 | accuracy: ↑0.01 | Helped |
| Use FuDD descriptions rather than the generic template set | CLIP ViT-L/14@336px · Stanford Cars | accuracy: 78.96 vs. 79.12 | Hurt |
| Replace differential descriptions with non-differential descriptions (ablation) | FuDD · DTD | accuracy: 56.91 vs. 56.44 | Hurt |
| Replace differential descriptions with non-differential descriptions (ablation) | FuDD · FGVCAircraft | accuracy: 39.06 vs. 36.90 | Hurt |
| Limit differential descriptions to k=10 ambiguous classes instead of k=|C| (ablation) | FuDD · CUB | accuracy: 65.90 vs. 66.03 | No clear effect |
| Fine-tune Llama 2 before generating FuDD descriptions (ablation) | Llama 2 FuDD · EuroSAT | accuracy: 37.78 -> 61.19 | Helped |
| Fine-tune Llama 2 before generating FuDD descriptions (ablation) | Llama 2 FuDD · Flowers102 | accuracy: 78.00 -> 77.51 | Hurt |
| Increase the number of described ambiguous classes from k=10 to k=|C| (ablation) | Llama 2 FuDD FT · Stanford Cars | accuracy: 78.20 -> 79.06 | Helped |
| Increase the number of described ambiguous classes from k=10 to k=|C| (ablation) | Llama 2 FuDD FT · Oxford Pets | accuracy: 92.67 -> 92.31 | Hurt |
Surprising findings
- Non-differential descriptions (attributes not separating ambiguous classes) lower accuracy relative to differential ones, showing not all semantic information helps.
- Naive LLM-generated descriptions perform worse than a generic template set on several datasets with the ViT-B/32 backbone.
Limitations
- Relies on proprietary GPT-3.5 API for best results; open Llama 2 model is weaker without fine-tuning.
- ImageNet experiments limited to cached pairwise descriptions for ambiguous classes detected by ViT-B/32, so the method is not fully per-image adaptive there.
- Description quality depends on LLM knowledge; many generated pairwise descriptions are incorrect or not useful for rare concepts (e.g., EuroSAT with base Llama 2).
Experimental setup
Datasets: CUB200-2011 DTD EuroSAT FGVCAircraft Flowers102 Food101 ImageNet ImageNetV2 Oxford Pets Places365 Stanford Cars Stanford Dogs
Benchmarks: CUB200-2011 DTD EuroSAT FGVCAircraft Flowers102 Food101 ImageNet ImageNetV2 Oxford Pets Places365 Stanford Cars Stanford Dogs
Metrics: accuracy average accuracy improvement
Baselines: Single Template Template Set Naive LLM LaBo CoOp SuS-X WaffleCLIP
Builds on: CLIP Menon & Vondrick naive LLM descriptions WaffleCLIP SuS-X
Closest papers by meaning
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.90)
- Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · similarity 0.90)
- Large Language Models are Good Prompt Learners for Low-Shot Image Classification (CVPR 2024 · similarity 0.88)
- Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.88)
- TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.87)
- Making Large Vision Language Models to Be Good Few-Shot Learners (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.