Research map / Vision-language and multimodal / CLIP and zero-shot
Tree of Attributes Prompt Learning for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Tree of Attributes Prompt learning (TAP), which distills a concept-attribute-description tree from GPT-3.5 and learns attribute-specific visual expert tokens and text prompts with a vision-conditional pooling layer for CLIP adaptation.
Problem
Prompt learning for vision-language models typically uses only class names or unstructured LLM-generated descriptions, which underuse the rich contextual knowledge and can include descriptions that are irrelevant to a given image.
Approach
Generate a structured concept-attribute-description tree from an LLM per dataset/class, learn one visual expert prompt token per attribute, and pool the descriptions per attribute with a cross-attention layer conditioned on the image, then fuse attribute-level logits with the CLS-token logit.
Key results
- Base-to-novel average harmonic mean of 81.04% vs 79.97% for PromptSRC across 11 datasets.
- Cross-dataset transfer average of 66.56% over 10 target datasets, versus 65.81% for PromptSRC.
- 16-shot few-shot classification average of 83.37% vs 82.87% for PromptSRC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TAP instead of PromptSRC | base-to-novel generalization · Base-to-novel generalization, average | HM: 79.97 -> 81.04 | Helped |
| use TAP instead of PSRC | cross-dataset transfer · Cross-dataset transfer, target average | accuracy: 65.81 -> 66.56 | Helped |
| use TAP instead of PSRC | 16-shot classification · 16-shot classification, average | accuracy: 82.87 -> 83.37 | Helped |
| replace an unstructured description set with the Tree of Attributes (ablation) | TAP · Base-to-novel generalization, average | HM: 2.11% | Helped |
| replace an unstructured description set with the Tree of Attributes (ablation) | TAP · Base-to-novel generalization, average | base accuracy: 1.86% | Helped |
| replace an unstructured description set with the Tree of Attributes (ablation) | TAP · Base-to-novel generalization, average | novel accuracy: 2.31% | Helped |
| add visual expert tokens for fine-grained attributes instead of relying on the CLS token (ablation) | TAP · Base-to-novel generalization, average | base accuracy: 0.89% | Helped |
| add visual expert tokens for fine-grained attributes instead of relying on the CLS token (ablation) | TAP · Base-to-novel generalization, average | novel accuracy: 0.78% | Helped |
| add visual expert tokens for fine-grained attributes instead of relying on the CLS token (ablation) | TAP · Base-to-novel generalization, average | HM: 0.82% | Helped |
| replace CLS-only prediction with fusion of CLS and expert-token predictions (ablation) | TAP · Base-to-novel generalization | accuracy | Helped |
| dynamically determine the attribute set per dataset instead of fixing the number of attributes (ablation) | TAP · Base-to-novel generalization | accuracy | Helped |
| increase the fixed attribute count from 1 to 7 (ablation) | TAP · Base-to-novel generalization | accuracy | Helped |
Surprising findings
- Replacing the GPT-3.5 descriptions with those from a locally served Qwen-2-7B-Instruct gives comparable results (HM 80.83 vs 81.04).
- The paper notes LLMs can generate identical descriptions for distinct fine-grained classes, hurting novel-class prediction.
Limitations
- Relies on LLM-generated descriptions, which can be identical for visually similar fine-grained classes.
- Per-dataset attribute sets and learning rate groupings require dataset-specific tuning, and the attribute set is fixed to 4 for cross-dataset transfer.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT
Metrics: accuracy harmonic mean
Baselines: CLIP CoOp Co-CoOp ProGrad RPO LoGoPrompt PromptSRC MaPLe
Builds on: CoOp CoCoOp PromptSRC VPT MaPLe
Closest papers by meaning
- Prompt Tuning In a Compact Attribute Space (AAAI 2025 · similarity 0.91)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.91)
- ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models (CVPR 2024 · similarity 0.90)
- Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models (AAAI 2024 · similarity 0.90)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · similarity 0.89)
- ProAPO: Progressively Automatic Prompt Optimization for Visual Classification (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.