Research map / Vision-language and multimodal / CLIP and zero-shot
JoAPR: Cleaning the Lens of Prompt Learning for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JoAPR, a label-noise-robust prompt learning framework for CLIP that uses GMM-based joint adaptive thresholds to partition clean/noisy few-shot data and refurbishes labels before retraining CoOp-style prompts.
Problem
Prompt learning for vision-language models (CoOp) on few-shot datasets degrades substantially under high-ratio and complex (pairflip) label noise, and prior work does not address this.
Approach
Partition samples with a two-component GMM over a loss that adds a confidence-compensation term, using two adaptive thresholds (one from GMM overlap, one from the previous epoch's clean fraction), then refurbish labels by blending original labels with model predictions weighted by clean posterior and softmax confidence, followed by MixMatch-based retraining with a uniform-prediction regularizer.
Key results
- On Caltech101 with 75% Pairflip noise, CoOp+JoAPR retains about 84% accuracy versus 16% for CoOp.
- On Food101N (real noise), CoOp reaches 69.50% while CoOp+JoAPR reaches 72.57% and CoOp+JoAPR* 73.87%.
- With 100% noise on Caltech101 and OxfordPets, CoOp+JoAPR reaches 81.50% and 72.60% (Symflip), where CoOp falls to 1.30% and 4.70%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply JoAPR label partitioning and refurbishment | CoOp · Food101N | accuracy: 69.50 -> 72.57 | Helped |
| apply JoAPR label partitioning and refurbishment | CoOp · ImageNet, Pairflip 75.0% | accuracy: 20.33 -> 53.67 | Helped |
| apply JoAPR label partitioning and refurbishment | CoOp · Caltech101, Pairflip 75.0% | accuracy: 16.23 -> 84.43 | Helped |
| remove Warmup (ablation) | JoAPR · EuroSAT, Pairflip 75.0% | accuracy: 39.47 -> 27.53 | Hurt |
| add threshold θ1 to joint adaptive partitioning (ablation) | JoAPR · EuroSAT, Pairflip 75.0% | accuracy: 20.07 -> 39.47 | Helped |
| add threshold θ2 to joint adaptive partitioning (ablation) | JoAPR · EuroSAT, Pairflip 75.0% | accuracy: 35.57 -> 39.47 | Helped |
| add psoftmax to clean-label confidence (ablation) | JoAPR · EuroSAT, Pairflip 75.0% | accuracy: 23.03 -> 39.47 | Helped |
| add clean-label refurbishment (ablation) | JoAPR · EuroSAT, Pairflip 75.0% | accuracy: 19.13 -> 39.47 | Helped |
| add noisy-label refurbishment (ablation) | JoAPR · EuroSAT, Pairflip 75.0% | accuracy: 19.00 -> 39.47 | Helped |
| add MixMatch (ablation) | JoAPR · EuroSAT, Pairflip 75.0% | accuracy: 10.53 -> 39.47 | Helped |
Surprising findings
- With 100% of training labels mislabeled, CoOp+JoAPR still achieves high accuracy on Caltech101 and OxfordPets, suggesting carefully annotated few-shot data may not always be required.
- Adding the confidence compensation term can increase overlap between clean and noisy loss distributions on larger few-shot datasets.
Limitations
- Gains are reduced or inconsistent on some settings, e.g., CoOp+JoAPR underperforms CoOp at 12.5% Symflip on ImageNet and the method's compensation term can hurt on larger datasets.
- Evaluated only with CLIP ResNet-50 and CoOp/CoCoOp-style prompt learning on 16-shot data.
- Relies on the assumption that misclassified noisy samples can still be partly refurbished; performance drops at 75% noise on EuroSAT for JoAPR.
Experimental setup
Datasets: ImageNet SUN397 Caltech101 Flowers102 StanfordCars OxfordPets UCF101 EuroSAT DTD Food101N
Benchmarks: ImageNet SUN397 Caltech101 Flowers102 StanfordCars OxfordPets UCF101 EuroSAT DTD Food101N
Metrics: accuracy
Baselines: CoOp CoOp+GCE CoOp+DivideMix CoCoOp
Builds on: CoOp DivideMix CLIP MixMatch Generalized Cross Entropy
Closest papers by meaning
- NLPrompt: Noise-Label Prompt Learning for Vision-Language Models (CVPR 2025 · similarity 0.91)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.88)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.88)
- Active Prompt Learning in Vision Language Models (CVPR 2024 · similarity 0.88)
- Consistency-guided Prompt Learning for Vision-Language Models (ICLR 2024 · similarity 0.88)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.