Research map / Vision-language and multimodal / CLIP and zero-shot
NLPrompt: Noise-Label Prompt Learning for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NLPrompt, which uses MAE loss for prompt learning on noisy-label data and a CLIP-text-initialized optimal transport (PromptOT) step to split data into clean (CE) and noisy (MAE) subsets.
Problem
Prompt learning for vision-language models such as CLIP degrades when training labels are noisy, and existing noisy-label methods for prompt learning remain limited.
Approach
Shows that MAE loss is robust for CLIP prompt learning (PromptMAE), supports this with a feature-learning theory argument, and uses text features as OT prototypes to purify the dataset, applying CE to the clean subset and MAE to the noisy subset.
Key results
- On Flowers102 with symmetric noise, NLPrompt reaches 93.87% at 12.5% noise versus 88.93% for CoOp, and 76.80% at 75% noise versus 37.17% for CoOp.
- On Food101N (real-world noisy labels), NLPrompt achieves 76.46% accuracy versus 72.57% for JoAPR and 69.50% for CoOp.
- NLPrompt improves VPT, MaPLe, and PromptSRC under symmetric noise on EuroSAT, e.g., VPT at 75% noise rises from 27.67% to 73.90%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MAE loss for all training data rather than CE loss (ablation) | Prompt learning on Flowers102 · Flowers102, symmetric noise | accuracy (Avg): 79.15 -> 85.90 | Helped |
| use CLIP text features instead of random prototype initialization (ablation) | PromptOT / NLPrompt · Flowers102, symmetric noise | accuracy (Avg): 80.94 -> 92.00 | Helped |
| add training on clean data with CE loss (ablation) | PromptOT / NLPrompt · Flowers102, symmetric noise | accuracy (Avg): 87.36 -> 92.00 | Helped |
| add training on noisy data with MAE loss (ablation) | PromptOT / NLPrompt · Flowers102, symmetric noise | accuracy (Avg): 82.16 -> 92.00 | Helped |
| remove noisy data from training (ablation) | NLPrompt · Flowers102, symmetric noise | accuracy (Avg): 92.00 -> 82.16 | Hurt |
| remove clean data from training (ablation) | NLPrompt · Flowers102, symmetric noise | accuracy (Avg): 92.00 -> 87.36 | Hurt |
| use NLPrompt rather than the strongest listed baseline | CoOp with CLIP · Flowers102, Sym 75.0% | accuracy: 70.37 -> 76.80 | Helped |
| use NLPrompt rather than the strongest listed baseline | CoOp with CLIP · Flowers102, Asym 75.0% | accuracy: 39.23 -> 55.33 | Helped |
| use NLPrompt rather than the strongest listed baseline | CoOp with CLIP · DTD, Sym 75.0% | accuracy: 33.67 -> 39.80 | Helped |
| use NLPrompt rather than the strongest listed baseline | CoOp with CLIP · OxfordPets, Sym 75.0% | accuracy: 74.40 -> 70.77 | Hurt |
| use NLPrompt rather than the strongest listed baseline | CoOp with CLIP · Food101N | accuracy: 72.57 -> 76.46 | Helped |
Surprising findings
- Plain MAE loss in prompt learning already substantially outperforms CE under heavy noise, and converges faster than expected for MAE.
- JoAPR underperforms the simpler GCE baseline on several asymmetric-noise settings at high noise rates.
Limitations
- Extension to unbalanced class distributions is left for future work.
- Experiments use only 16-shot training sets and synthetic noise on most datasets.
- Theoretical analysis is restricted to a binary classification setting with simplified linear feature assumptions.
Experimental setup
Datasets: Caltech101 DTD EuroSAT Flowers102 OxfordPets StanfordCars UCF101 Food101N
Benchmarks: Caltech101 DTD EuroSAT Flowers102 OxfordPets StanfordCars UCF101 Food101N
Metrics: test accuracy
Baselines: CoOp CoOp+GCE JoAPR
Builds on: CoOp CLIP JoAPR Sinkhorn optimal transport feature learning theory
Closest papers by meaning
- JoAPR: Cleaning the Lens of Prompt Learning for Vision-Language Models (CVPR 2024 · similarity 0.91)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · similarity 0.90)
- Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation (ICLR 2025 · similarity 0.89)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.89)
- Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior (AAAI 2024 · similarity 0.89)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.