Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning

ProTeCt: Prompt Tuning for Taxonomic Open Set Classification

CVPR 2024 · Label-efficient and robust learning › Zero-/few-shot and meta-learning · approach: Robust fine-tuning of zero-shot models · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ProTeCt, a prompt-tuning add-on (dynamic treecut loss plus node-centric loss) that makes CLIP prompt learners hierarchically consistent for taxonomic open set classification, with new HCA and MTA metrics.

Problem

Few-shot prompt-tuned vision-language models like CLIP give correct leaf predictions but inconsistent predictions at coarser taxonomy levels, failing the taxonomic open set (TOS) setting where label sets can come from any level of a class hierarchy.

Approach

Train prompts with two regularizers: a node-centric cross-entropy over each internal node's children, and a dynamic treecut loss that samples random tree cuts (via an efficient mask-based sampler) as label sets per example, added to existing prompt-tuning losses while keeping encoders frozen.

Key results

• On 16-shot CoOp with ImageNet, HCA rises from 2.99 to 37.74 and MTA(25) from 46.98 to 88.61, with leaf accuracy essentially unchanged (71.23 to 69.92).
• On CIFAR-100 16-shot CoOp, ProTeCt raises HCA from 10.04 to 56.85 while leaf accuracy stays at about 72.9.
• ProTeCt gains transfer zero-shot from ImageNet to ImageNetV2, -Sketch, -A and -R, e.g. HCA gains of 12.7 to 30.5 points on 16-shot CoOp/MaPLe.

Measured effects reported in the paper

Surprising findings

• Node-centric loss alone can degrade leaf accuracy (e.g., 64.20 leaf accuracy on CIFAR-100 16-shot CoOp without DTL), and the two losses are complementary.

Limitations

• Full treecut space is intractable, so MTA is estimated from 25 sampled treecuts.
• Leaf accuracy can drop slightly on ImageNet variants and some settings (e.g., -1.31 on ImageNet 16-shot CoOp).
• Hierarchies rely on WordNet or dataset-provided taxonomies; SUN hierarchy required pruning 73 leaf classes.

Experimental setup

Datasets: CIFAR-100 SUN ImageNet ImageNetV2 ImageNet-Sketch ImageNet-A ImageNet-R

Benchmarks: CIFAR-100 SUN ImageNet ImageNetV2 ImageNet-Sketch ImageNet-A ImageNet-R

Metrics: leaf accuracy Hierarchical Consistent Accuracy (HCA) Mean Treecut Accuracy (MTA)

Baselines: CLIP CoOp MaPLe CLIP-Adapter CLIP+LoRA

Builds on: CoOp MaPLe CLIP CoCoOp CLIP-Adapter

Closest papers by meaning

• Compound Text-Guided Prompt Tuning via Image-Adaptive Cues (AAAI 2024 · similarity 0.85)
• OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning (ICML 2025 · similarity 0.84)
• Tree of Attributes Prompt Learning for Vision-Language Models (ICLR 2025 · similarity 0.84)
• ProAPO: Progressively Automatic Prompt Optimization for Visual Classification (CVPR 2025 · similarity 0.84)
• Taxonomy-Aware Evaluation of Vision-Language Models (CVPR 2025 · similarity 0.84)
• Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · similarity 0.84)