Research map / Vision-language and multimodal / CLIP and zero-shot
Active Prompt Learning in Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PCB, a pseudo-class-balanced active learning sampler for prompt learning (CoOp) on CLIP, which uses VLM pseudo-labels to balance the labeled set and improve over conventional active learning and random sampling.
Problem
Conventional active learning applied to prompt-learning adaptation of vision-language models like CLIP can underperform random sampling because the selected labeling candidates are class-imbalanced, inheriting the VLM's uneven per-class knowledge.
Approach
Two-stage selection: first pick an informative subset with a conventional AL method, pseudo-label it with the VLM, then build the query set by class-balanced sampling from the pseudo-labeled pool so each class is equally represented; optionally augment class names with LLM-generated descriptions (AS/AE).
Key results
- PCB (AS) with BADGE reaches 74.36% average final accuracy across seven datasets with ViT-B/32, versus 72.43% for BADGE alone and 70.54% for random sampling.
- PCB improves average accuracy with Entropy, Coreset, and BADGE; on average across datasets the gain is up to 4.64%.
- Conventional Entropy and Coreset underperform random sampling on EuroSAT and Aircraft, attributed to class-imbalanced selections.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply PCB with average similarity to BADGE | BADGE active prompt learning · Average across seven datasets | Avg Acc: 72.43 -> 74.36 | Helped |
| Apply PCB with average similarity to BADGE | BADGE active prompt learning · Flowers102 | accuracy: 96.44±0.16 -> 96.71±0.29 | Helped |
| Apply PCB with average similarity to BADGE | BADGE active prompt learning · DTD | accuracy: 61.52±1.25 -> 62.33±1.06 | Helped |
| Apply PCB with average similarity to BADGE | BADGE active prompt learning · Oxford Pets | accuracy: 82.33±0.72 -> 83.16±0.18 | Helped |
| Apply PCB with average similarity to BADGE | BADGE active prompt learning · EuroSAT | accuracy: 81.66±0.41 -> 81.50±1.11 | No clear effect |
| Apply PCB with average similarity to BADGE | BADGE active prompt learning · Caltech101 | accuracy: 93.79±0.25 -> 93.85±0.37 | No clear effect |
| Apply PCB with average similarity to BADGE | BADGE active prompt learning · Stanford Cars | accuracy: 70.56±0.31 -> 70.70±0.79 | No clear effect |
| Apply PCB with average similarity to BADGE | BADGE active prompt learning · Aircraft | accuracy: 31.79±0.74 -> 32.27±0.66 | Helped |
| Add PCB to BADGE without description augmentation (ablation) | BADGE active prompt learning · Average across seven datasets | Avg Acc: 72.43 -> 72.95 | Helped |
| Use average similarity rather than average embedding with PCB (ablation) | BADGE + PCB · Average across seven datasets | Avg Acc: 73.27 -> 74.36 | Helped |
| Use description augmentation with average similarity, without PCB (ablation) | BADGE active prompt learning · Average across seven datasets | Avg Acc: 72.43 -> 74.01 | Helped |
| Increase the informative candidate-pool fraction γ (ablation) | PCB with Coreset · Flowers102 | accuracy | Helped |
Surprising findings
- Naively applying active learning to VLM prompt learning can perform worse than random sampling on some datasets (e.g., EuroSAT, Aircraft).
- On Oxford Pets, PCB with ViT-B/32 underperforms zero-shot CLIP, which the authors attribute to too few labeled samples.
Limitations
- Oxford Pets results remain below zero-shot CLIP at the default budget of N=K per round.
- Evaluated only on image classification datasets with CoOp-style prompt learning and CLIP encoders.
Experimental setup
Datasets: EuroSAT Oxford Pets DTD Caltech101 Flowers102 Stanford Cars FGVC-Aircraft
Benchmarks: Flowers102 DTD Oxford Pets EuroSAT Caltech101 Stanford Cars FGVC-Aircraft
Metrics: final accuracy average accuracy class-count variance (imbalance)
Baselines: Random sampling Entropy Coreset BADGE CLIP zero-shot CoCoOp MaPLe Full data
Builds on: CoOp BADGE CLIP CuPL Entropy-based active learning
Closest papers by meaning
- Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.90)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.88)
- Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data (ICML 2024 · similarity 0.88)
- JoAPR: Cleaning the Lens of Prompt Learning for Vision-Language Models (CVPR 2024 · similarity 0.88)
- Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves (CVPR 2025 · similarity 0.88)
- Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.