Research map / Vision-language and multimodal / CLIP and zero-shot
PromptKD: Unsupervised Prompt Distillation for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PromptKD, a two-stage unsupervised prompt distillation framework that transfers a ViT-L/14 CLIP teacher to a ViT-B/16 student via learnable visual prompts trained on unlabeled domain images, reusing pre-stored teacher text features as class vectors.
Problem
Prompt learning for CLIP relies on scarce labeled domain data; distilling knowledge from large CLIP teachers into lightweight students usually requires full fine-tuning and labeled or large unlabeled data with costly text-branch computation.
Approach
Pre-compute and store teacher text features once as shared class vectors, then train only the student's visual prompts and a projector (student image encoder frozen) by KL-divergence logit distillation from the teacher on unlabeled domain images, exploiting CLIP's decoupled modalities for transductive zero-shot inference.
Key results
- Average base-to-novel harmonic mean of 83.73 on 11 datasets with ViT-B/16, +3.76 over PromptSRC (base +2.70, novel +4.63).
- Cross-dataset transductive evaluation: average 71.33 vs 65.81 for PromptSRC (+5.52), better on 9 of 10 target datasets.
- On Flowers102, PromptKD reaches 90.24 HM vs 85.95 for PromptSRC and 86.87 for FPL, using unlabeled data only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill with PromptKD rather than PromptSRC | ViT-B/16 on the 11-dataset base-to-novel average · 11-dataset average, base-to-novel | HM: 79.97 -> 83.73 | Helped |
| Use logit-based KL distillation rather than feature-based distillation (ablation) | PromptKD on ImageNet · ImageNet | HM: 69.35 -> 76.22 | Helped |
| Add learnable visual prompts (ablation) | PromptKD on ImageNet · ImageNet | HM: 75.53 -> 76.22 | Helped |
| Share the teacher's pre-stored text features with the student (ablation) | PromptKD on ImageNet · ImageNet | HM: 75.98 -> 76.22 | Helped |
| Pre-train the teacher with PromptSRC rather than vanilla CLIP (ablation) | PromptKD student on ImageNet · ImageNet | HM: 74.50 -> 76.22 | Helped |
| Increase unlabeled distillation data from 16 to 64 images per category (ablation) | PromptKD on ImageNet · ImageNet | HM: 74.15 -> 76.22 | Helped |
| Distill from larger-capacity teacher image encoders (ablation) | PromptKD student on base-to-novel evaluation · ImageNet | HM: +2.22 | Helped |
Surprising findings
- Full fine-tuning of the student with distillation performed worse (HM 73.34) than prompt-only distillation with a projector (HM 76.22).
- Feature-level distillation (L1/MSE) underperformed logit-level KL distillation.
Limitations
- Distillation quality depends on whether unlabeled data represents the target domain; missing representation may weaken student performance.
- Teacher performance bounds student performance; pre-training of teacher requires labeled few-shot data.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101
Metrics: base accuracy novel accuracy harmonic mean accuracy GFLOPs FPS
Baselines: CoOp CoCoOp MaPLe PromptSRC CLIP-PR UPL LaFTer FPL IFPL GRIP CLIP-KD
Builds on: CLIP CoOp MaPLe PromptSRC CLIP-KD
Closest papers by meaning
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.93)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.91)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.91)
- Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior (AAAI 2024 · similarity 0.90)
- Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.