Research map / Vision-language and multimodal / CLIP and zero-shot
Task-Aware Clustering for Prompting Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Task-Aware Clustering (TAC), which clusters CLIP class embeddings with K-Means into a few centers, linearly maps them into pre-context tokens for learnable prompts, masks visual pre-context from patch tokens, and adds CLIP-consistency self-regulation for prompt learning.
Problem
Prompt learning for vision-language models with only a few labeled images per class overfits, yielding prompts that generalize poorly and have limited task-awareness.
Approach
Instead of adding constraints, inject task-aware pre-context built from K-Means cluster centers of CLIP class embeddings (only K=5 tokens rather than one per class), let learnable prompts interact with these tokens through the frozen encoders, mask patch-to-pre-context attention, and regularize prompts toward CLIP's own embeddings.
Key results
- Base-to-novel average harmonic mean of 81.24% vs. 80.48% for CoPrompt and 79.97% for PromptSRC across 11 datasets.
- Domain generalization average 61.63% over ImageNet variants, best among compared methods (ArGue 60.67%, PromptSRC 60.65%).
- Cross-dataset transfer average 66.53% vs. 67.00% for CoPrompt, so TAC is second-best on average; it gets the best ImageNet source accuracy (72.77%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply TAC instead of the strongest listed baseline on the base-to-novel average | CLIP ViT-B/16 · Base-to-novel average | HM: 80.78 -> 81.24 | Helped |
| apply TAC instead of CoPrompt | CLIP ViT-B/16 · Cross-dataset transfer average | Average accuracy: 67.00 -> 66.53 | Hurt |
| apply TAC instead of ArGue | CLIP ViT-B/16 · Domain generalization average | Average accuracy: 60.67 -> 61.63 | Helped |
| add textual task-aware pre-context (ablation) | TAC · Base-to-novel average | HM: 80.13 -> 81.24 | Helped |
| add visual task-aware pre-context (ablation) | TAC · Base-to-novel average | HM: 80.32 -> 81.24 | Helped |
| use K = 5 clustering centers instead of K = 0 (ablation) | TAC · Base-to-novel average | HM: 79.86 -> 81.24 | Helped |
| use K = 5 instead of K = 1 clustering centers (ablation) | TAC · Base-to-novel average | HM: 80.99 -> 81.24 | Helped |
| use K = 5 instead of K = 2 clustering centers (ablation) | TAC · Base-to-novel average | HM: 81.00 -> 81.24 | Helped |
| use K = 5 instead of K = 3 clustering centers (ablation) | TAC · Base-to-novel average | HM: 81.14 -> 81.24 | Helped |
| use K = 5 instead of K = 4 clustering centers (ablation) | TAC · Base-to-novel average | HM: 81.17 -> 81.24 | Helped |
| add textual and visual self-regulation (ablation) | TAC · Base-to-novel average | HM: 79.34 -> 81.24 | Helped |
| add both textual and visual consistency rather than textual consistency alone (ablation) | TAC · Base-to-novel average | HM: 78.32 -> 81.24 | Helped |
Surprising findings
- Cross-dataset transfer average is slightly below CoPrompt (66.53 vs 67.00), so the method is not uniformly best.
Limitations
- Cross-dataset transfer average is slightly behind CoPrompt.
- Gains in ablations are modest (e.g., masking adds only 0.41 HM points).
- Relies on frozen CLIP encoders and few-shot (16-shot) setting; generality to other backbones not shown in main text.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNet-A
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNet-A
Metrics: accuracy harmonic mean
Baselines: CoOp Co-CoOp MaPLe PromptSRC TCP CoPrompt ArGue RPO KgCoOp ProGrad
Builds on: CoOp PromptSRC CLIP MaPLe
Closest papers by meaning
- Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models (AAAI 2024 · similarity 0.89)
- Joint Scheduling of Causal Prompts and Tasks for Multi-Task Learning (CVPR 2025 · similarity 0.89)
- Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior (AAAI 2024 · similarity 0.89)
- Prompt Learning via Meta-Regularization (CVPR 2024 · similarity 0.88)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.88)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.