Research map / Vision-language and multimodal / CLIP and zero-shot
Compound Text-Guided Prompt Tuning via Image-Adaptive Cues
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TGP-T, a prompt tuning method for CLIP that supervises image-conditioned prompts with compound category-wise and content-wise text in vocabulary space, so only two prompt inputs go to the text encoder, cutting GPU memory by 93%.
Problem
Prompt tuning for CLIP (e.g., CoOp, CoCoOp) feeds N learnable category prompts into the text encoder, causing GPU memory to grow with the number of categories; prompts also embed category names, which hurts performance on ambiguous names.
Approach
Relocate category-center learning after the text encoder: a Bonder (transformer with cross-attention to image features) generates a small set of image-adaptive prompts, supervised by offline-generated category-wise templates and MiniGPT-4 content descriptions via cross-entropy in the vocabulary space, so no category names are needed at inference.
Key results
- TGP-T reaches 82.65% average accuracy over 11 datasets at 16 shots vs. 78.43% for CoCoOp and 80.75% for Cross-Modal Adaptation (ViT-B/16); TGP-T-F reaches 84.06%.
- On 16-shot ImageNet, TGP-T uses about 1 GB GPU memory vs. 18 GB for CoOp/CoCoOp, a 93% reduction, with 73.48% accuracy.
- On 16-shot FGVCAircraft, compound text supervision improves accuracy by 9.76% over the baseline without supervision (52.39 vs. 42.63).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TGP-T instead of CoCoOp | few-shot recognition · 11-dataset average, 16-shot | top-1 accuracy: 78.43% to 82.65% | Helped |
| Add LoRA to TGP-T | TGP-T · 11-dataset few-shot average, 16-shot | top-1 accuracy: 82.65 -> 84.06 | Helped |
| Use category-wise and content-wise text supervision together (ablation) | TGP-T · FGVCAircraft, 16-shot | accuracy: +9.76% | Helped |
| Use category-wise and content-wise text supervision together (ablation) | TGP-T · 11-dataset average, 1-shot | top-1 accuracy: ($\uparrow$2.84) | Helped |
| Use category-wise and content-wise text supervision together (ablation) | TGP-T · 11-dataset average, 16-shot | top-1 accuracy: ($\uparrow$1.46) | Helped |
| Use embedding-space rather than vocabulary-space supervision (ablation) | TGP-T · 11-dataset average, 16-shot | top-1 accuracy: 82.65 vs. 81.46 | Hurt |
| Use latent-space rather than vocabulary-space supervision (ablation) | TGP-T · 11-dataset average, 16-shot | top-1 accuracy: 82.65 vs. 81.58 | Hurt |
| Use Cross-Attention rather than Meta-Net in Bonder (ablation) | TGP-T · ImageNet, 16-shot | accuracy: 72.67 -> 73.48 | Helped |
| Increase Bonder depth from 1 to 8 (ablation) | TGP-T · ImageNet, 16-shot | accuracy: 73.48 -> 72.91 | Hurt |
| Tune attention as well as MLP LoRA layers (ablation) | TGP-T · ImageNet, 16-shot | accuracy: 74.39 -> 71.07 | Hurt |
| Tune MLP LoRA layers in both encoders rather than vision encoder only (ablation) | TGP-T · ImageNet, 16-shot | accuracy: 74.22 -> 74.39 | Helped |
| Tune MLP LoRA layers in both encoders rather than text encoder only (ablation) | TGP-T · ImageNet, 16-shot | accuracy: 73.57 -> 74.39 | Helped |
Surprising findings
- Category-wise or content-wise supervision alone yields little gain at higher shot counts; only the combination gives notable improvement.
- Deeper Bonder (depth 8) underperforms across all shot settings, suggesting overfitting or optimization difficulty.
Limitations
- Content-wise descriptions require offline generation with a large VLM (MiniGPT-4 on Vicuna-7B), adding preprocessing cost.
- Standalone category-wise or content-wise supervision gives limited gains at higher shot counts.
- Evaluation is limited to few-shot classification and ImageNet distribution shift with ViT-B/16 mainly.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets Flowers102 Food101 StanfordCars FGVCAircraft EuroSAT UCF101 DTD SUN397
Benchmarks: ImageNet Caltech101 OxfordPets Flowers102 Food101 StanfordCars FGVCAircraft EuroSAT UCF101 DTD SUN397 ImageNet-V2
Metrics: top-1 accuracy GPU memory (GB) training time
Baselines: Linear Probe CLIP CoOp CoCoOp Tip-Adapter Tip-Adapter-F MaPLe Cross-Modal Adaptation
Builds on: CLIP CoOp CoCoOp Tip-Adapter LoRA
Closest papers by meaning
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.91)
- Concept-Guided Prompt Learning for Generalization in Vision-Language Models (AAAI 2024 · similarity 0.90)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.89)
- Amend to Alignment: Decoupled Prompt Tuning for Mitigating Spurious Correlation in Vision-Language Models (ICML 2024 · similarity 0.89)
- ProAPO: Progressively Automatic Prompt Optimization for Visual Classification (CVPR 2025 · similarity 0.89)
- TextRefiner: Internal Visual Feature as Efficient Refiner for Vision-Language Models Prompt Tuning (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.