Research map / Vision-language and multimodal / CLIP and zero-shot
TCP:Textual-based Class-aware Prompt tuning for Visual-Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TCP, a textual prompt-tuning method that maps frozen CLIP class-name text embeddings into class-aware prompt tokens via a small Textual Knowledge Embedding module, inserted mid-layer into the CLIP text encoder.
Problem
CoOp-style prompt tuning of vision-language models uses domain-shared or image-conditional learnable tokens that do not model the class distribution, so they generalize poorly to unseen classes.
Approach
Derive class-aware prompt tokens from frozen CLIP class-level text embeddings through a down/up projection (TKE) and insert them into the 8th layer of the text encoder alongside shared learnable tokens; TKE is plug-and-play for other CoOp-based methods.
Key results
- Base-to-new generalization: TCP reaches 84.13 base / 75.36 new / 79.51 H average, versus KgCoOp 80.73/73.6/77.0 and PromptSRC 82.07/74.83/78.03 (re-implemented).
- Cross-dataset generalization from ImageNet: TCP averages 66.29%, the best among textual prompt tuning methods (DePT 65.55%), comparable to DAPT at 66.31%.
- Plugging TKE into KgCoOp raises base-to-new H from 77.00 to 79.51; into CoOp raises H from 74.6 to 76.09.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add class-aware TKE prompts to KgCoOp (ablation) | KgCoOp · Base-to-new generalization, Average | H: 77.00 -> 79.51 | Helped |
| add TKE (ablation) | CoOp · Base-to-new generalization, Average | H: 74.6 -> 76.09 | Helped |
| add TKE (ablation) | ProGrad · Base-to-new generalization, Average | H: 76.16 -> 77.46 | Helped |
| add TKE (ablation) | PromptSRC · Base-to-new generalization, Average | H: 78.03 -> 79.60 | Helped |
| add TKE (ablation) | DAPT · Base-to-new generalization, Average | H: 75.59 -> 79.24 | Helped |
| use class-aware prompt rather than domain-shared prompt (ablation) | KgCoOp prompt tuning · Base-to-new generalization, Average | H: 77 -> 79.36 | Helped |
| combine domain-shared and class-aware prompts (ablation) | TCP · Base-to-new generalization, Average | H: 79.36 -> 79.51 | Helped |
| replace random prompt initialization with handcrafted template 'a photo of a' (ablation) | TCP · Base-to-new generalization, Average | H: 79.51 -> 79.18 | Hurt |
| replace random prompt initialization with 'this is a picture' (ablation) | TCP · Base-to-new generalization, Average | H: 79.51 -> 79.06 | Hurt |
| insert class-aware prompts at layers 4 and 8 rather than layer 8 only (ablation) | TCP · Base-to-new generalization, Average | H: 79.51 -> 79.33 | No clear effect |
| insert class-aware prompts at layers 4, 8 and 10 rather than layer 8 only (ablation) | TCP · Base-to-new generalization, Average | H: 79.51 -> 79.02 | Hurt |
| use more prompt insertion layers, at 4, 6, 8 and 10 (ablation) | TCP · Base-to-new generalization, Average | H: 79.51 -> 79.13 | Hurt |
Surprising findings
- Random-initialized 'X X X X' template outperforms the handcrafted 'a photo of a' template for TCP.
- Shallow single-layer class-aware prompt outperforms deeper multi-layer variants.
Limitations
- Class-aware prompt depends on discriminative ability of the general CLIP textual embedding; weak textual embeddings yield weaker classifiers.
- Performs poorly on FGVCAircraft dataset.
- Base-to-new and few-shot results rely on CLIP ViT-B/16 only in the reported setup.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101
Metrics: accuracy harmonic mean (H)
Baselines: CoOp CoCoOp KgCoOp ProGrad ProDA RPO PLOT MaPLe PromptSRC DePT DAPT TaskRes
Builds on: CoOp KgCoOp CLIP CoCoOp
Closest papers by meaning
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.87)
- ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models (CVPR 2024 · similarity 0.86)
- All You Need is One: Capsule Prompt Tuning with a Single Vector (NeurIPS 2025 · similarity 0.85)
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.85)
- VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion (NeurIPS 2025 · similarity 0.85)
- Prompt Tuning In a Compact Attribute Space (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.