Research map / Vision-language and multimodal / CLIP and zero-shot
Prompt Tuning In a Compact Attribute Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PTinCAS, which builds a compact attribute vocabulary from LLM-generated class descriptions and uses attribute supervision plus object-aware visual prompting to improve CLIP prompt tuning.
Problem
Prompt tuning of vision-language models like CLIP is driven only by class-name supervision, which limits fine-grained perception and generalization to unseen classes and base-class overfitting.
Approach
Builds a compact attribute space by clustering GPT-3.5-generated descriptive attributes with CLIP text embeddings (k-means, K=64), then trains soft prompts jointly on class naming (cross-entropy) and multi-label attribute prediction, with a self-contained GradCAM-based background-grayscaling visual prompt and momentum-updated attribute prototypes.
Key results
- Base-to-novel: PTinCAS improves CoOp HM by 2.66% (71.66 to 74.32), and novel-class accuracy by 4.10%, averaged over 11 datasets.
- Base-to-novel HM with MaPLe increases from 78.55 to 79.38 and with PromptSRC from 79.97 to 80.72.
- Cross-domain: PTinCAS on MaPLe raises average target accuracy from 60.27 to 60.84 across four ImageNet variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PTinCAS attribute-based learning | PromptSRC · Base-to-novel generalization (11 datasets average) | HM: 79.97 -> 80.72 | Helped |
| Add PTinCAS attribute-based learning | PromptSRC · Cross-domain generalization, ImageNet source | Avg. accuracy: 60.65 -> 60.88 | Helped |
| Add PTinCAS attribute-based learning | PromptSRC · Few-shot classification (11 datasets average) | accuracy: 72.32 -> 72.93; 75.29 -> 76.14; 78.35 -> 78.94; 80.69 -> 81.12; 82.87 -> 83.33 | Helped |
| Remove attribute loss LATT (λ=0) (ablation) | MaPLe + PTinCAS · Base-to-novel generalization (11 datasets average) | HM: 78.55 -> 79.38 | Hurt |
| Set attribute-loss coefficient λ=1 instead of λ=2 (ablation) | MaPLe + PTinCAS · Base-to-novel generalization (11 datasets average) | HM: 78.97 -> 79.38 | Hurt |
| Set attribute-loss coefficient λ=3 instead of λ=2 (ablation) | MaPLe + PTinCAS · Base-to-novel generalization (11 datasets average) | HM: 79.38 -> 79.16 | Hurt |
| Use the original image instead of object-aware visual prompting (ablation) | MaPLe + PTinCAS · Base-to-novel generalization (11 datasets average) | HM: 78.75 -> 79.38 | Hurt |
| Set background pixels to zero (ablation) | MaPLe + PTinCAS · Base-to-novel generalization (11 datasets average) | HM: 78.35 -> 79.38 | Hurt |
| Blur the background with a 21×21 Gaussian kernel instead of using OVP (ablation) | MaPLe + PTinCAS · Base-to-novel generalization (11 datasets average) | HM: 79.13 -> 79.38 | Hurt |
| Remove clustering when constructing the attribute space (ablation) | MaPLe + PTinCAS · Base-to-novel generalization (11 datasets average) | HM: 64.90 -> 79.38 | Hurt |
| Set attribute clustering number K=16 instead of K=64 (ablation) | MaPLe + PTinCAS · Base-to-novel generalization (11 datasets average) | HM: 78.91 -> 79.38 | Hurt |
| Set attribute clustering number K=32 instead of K=64 (ablation) | MaPLe + PTinCAS · Base-to-novel generalization (11 datasets average) | HM: 79.17 -> 79.38 | Hurt |
Surprising findings
- Without clustering (raw attributes), base-to-novel HM drops to 64.90, far below the 78.55 MaPLe baseline.
Limitations
- Relies on LLM-generated attributes that the authors note can be noisy and per-class generated.
- Attribute prototypes depend on a fixed number of clusters K (chosen as 64) and a loss weight λ tuned on the benchmarks.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNetV2
Metrics: accuracy harmonic mean
Baselines: CoOp CoCoOp MaPLe PromptSRC ArGue DePT CPT FGVP
Builds on: CoOp CoCoOp MaPLe PromptSRC Grad-CAM Visual Classification via Description from LLMs
Closest papers by meaning
- ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models (CVPR 2024 · similarity 0.93)
- Tree of Attributes Prompt Learning for Vision-Language Models (ICLR 2025 · similarity 0.91)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.90)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.90)
- VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion (NeurIPS 2025 · similarity 0.89)
- Concept-Guided Prompt Learning for Generalization in Vision-Language Models (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.