Research map / Vision-language and multimodal / CLIP and zero-shot
Enhancing Target-unspecific Tasks through a Features Matrix
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Features Matrix (FM) built from frozen CLIP text features of many hand-crafted prompts, plugged into CoOp, CoCoOp, MaPLe and PromptSRC with an extra contrastive loss, to improve base-to-novel, domain and cross-dataset generalization.
Problem
Prompt learning methods for CLIP-like VLMs overfit to target-specific data and lose general knowledge, performing poorly on target-unspecific (novel-class, domain-shift, cross-dataset) tasks.
Approach
Extract general knowledge as a frozen matrix of text features from many hand-crafted prompt templates combined with class names, and align tuned visual features to the 'unexpected' (low-scoring designated and high-scoring non-designated) pre-trained text features via a contrastive loss added to the standard cross-entropy objective.
Key results
- Integrated with PromptSRC, the method reaches 85.70% base and 77.35% novel average over 11 datasets (HM 81.32%), versus PromptSRC's 84.26/76.10 (HM 79.97).
- On domain generalization with PromptSRC backbone, average over ImageNet variants is 61.92% vs 60.65% for PromptSRC.
- Cross-dataset generalization with PromptSRC backbone averages 67.62% over 10 unseen datasets vs 65.81% for PromptSRC; the method is lower than DePT on ImageNet source training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Features Matrix to PromptSRC | PromptSRC with CLIP ViT-B/16 · Base-to-novel generalization, average over 11 datasets | Novel accuracy: 76.10 -> 77.35 | Helped |
| Add Features Matrix to PromptSRC | PromptSRC with CLIP ViT-B/16 · Base-to-novel generalization, average over 11 datasets | HM: 79.97 -> 81.32 | Helped |
| Add Features Matrix to PromptSRC | PromptSRC with CLIP ViT-B/16 · Domain generalization, ImageNet variants average | Accuracy: 60.65 -> 61.92 | Helped |
| Add Features Matrix to PromptSRC | PromptSRC with CLIP ViT-B/16 · Cross-dataset generalization, average over 10 target datasets | Accuracy: 65.81 -> 67.62 | Helped |
| Increase learning depth from 9 to 11 (ablation) | PromptSRC + FM · Base-to-novel generalization, average over 11 datasets | HM: 81.32 -> 80.21 | Hurt |
| Set prompt length to 4 rather than 1, 2, 6, 8, or 10 (ablation) | PromptSRC + FM · Base-to-novel generalization, average over 11 datasets | HM: 77.01; 80.05; 81.32; 80.30; 79.11; 78.54 | Mixed |
| Set gamma to 0.1 rather than 0.05, 0.5, or 0.9 (ablation) | PromptSRC + FM · Base-to-novel generalization, average over 11 datasets | HM: 80.13; 81.32; 81.00; 79.88 | Mixed |
| Set top-beta and low-beta to 5 rather than 3, 4, or 6 (ablation) | PromptSRC + FM · Base-to-novel generalization, average over 11 datasets | HM: 78.21; 79.81; 81.32; 80.51 | Mixed |
| Use ViT-L/14 rather than ViT-B/16 (ablation) | PromptSRC + FM · Average over 11 datasets | HM: 81.32 -> 84.20 | Helped |
| Use ViT-B/32 rather than ViT-B/16 (ablation) | PromptSRC + FM · Average over 11 datasets | HM: 81.32 -> 79.70 | Hurt |
Surprising findings
- On the ImageNet source training in the cross-dataset and domain settings, the method underperforms DePT on the source accuracy (71.41 vs 71.60 for PromptSRC-based variants, and 71.82 vs 72.63 for CoOp-based).
- Per-dataset results show the method is sometimes worse than baselines on base accuracy, e.g., StanfordCars and DTD base accuracy with CoOp and CoCoOp backbones.
Limitations
- Training is slower because of multiple processes in the method.
- ImageNet source-domain accuracy is lower than DePT and some baselines.
- Several hyperparameters (gamma, top/low beta, prompt length, depth) were tuned on the evaluation setting.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNet-V2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNet-V2
Metrics: accuracy harmonic mean
Baselines: CLIP CoOp CoCoOp ProDA PLOT ProGrad KgCoOp DePT MaPLe PromptSRC
Builds on: CoOp CoCoOp MaPLe PromptSRC KgCoOp
Closest papers by meaning
- Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP (NeurIPS 2024 · similarity 0.89)
- Prompt Learning via Meta-Regularization (CVPR 2024 · similarity 0.89)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models (CVPR 2024 · similarity 0.88)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.88)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.