Research map / Vision-language and multimodal / CLIP and zero-shot
Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?
Cheng-En Wu, Yu Tian, Haichao Yu, Heng Wang, Pedro Morgado, Yu Hen Hu, Linjie Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| keep CLIP text encoder fixed instead of fine-tuning it (ablation) | CLIP prompt tuning · OxfordPets 16-shot, 50% label noise | accuracy (%): 41.21 -> 73.13 | Improved |
| initialize classifier from CLIP text embeddings instead of random (ablation) | linear probe · OxfordPets 16-shot, 50% label noise | accuracy (%): 36.63 -> 44.21 | Improved |
| use prompt tuning through frozen text encoder instead of linear classifier | CLIP · Food101 16-shot, 50% label noise | accuracy (%): 34.37 -> 64.30 | Improved |
| make the classname token learnable (Full Prompt Tuning) (ablation) | CLIP prompt tuning · OxfordPets 16-shot, 50% label noise | accuracy (%): 73.13 -> 50.50 | Hurt |
| make classname token learnable at zero noise (ablation) | CLIP prompt tuning · OxfordPets and Food101 16-shot, 0% noise | accuracy (%): 87.89 -> 85.39 (OxfordPets) | Hurt |
| use confusion noise instead of random noise at 50% rate | prompt tuning · OxfordPets 16-shot, 50% label noise | accuracy (%): 73.13 -> 66.55 | Hurt |
| use ViT-B/32 image encoder instead of RN50 | prompt tuning · ImageNet 16-shot, 50% label noise | accuracy (%): 57.97 -> 61.75 | Improved |
| zero learnable context tokens (Ctx-0) under 50% noise (ablation) | prompt tuning · context length study | accuracy | Mixed |
| random sampling of pseudo-labels with GCE loss instead of top-confidence selection | UPL · ImageNet unsupervised prompt tuning | accuracy (%): 60.22 -> 62.14 | Improved |
| replace CE with GCE in Robust UPL (ablation) | Robust UPL · DTD unsupervised prompt tuning | accuracy (%): 44.80 -> 46.69 | Improved |
| Robust UPL versus UPL on Caltech101 | Robust UPL · Caltech101 unsupervised prompt tuning | accuracy (%): 90.10 -> 88.07 | Hurt |
Closest papers by meaning
- NLPrompt: Noise-Label Prompt Learning for Vision-Language Models (CVPR 2025 · similarity 0.92)
- Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models (ICML 2025 · similarity 0.90)
- Amend to Alignment: Decoupled Prompt Tuning for Mitigating Spurious Correlation in Vision-Language Models (ICML 2024 · similarity 0.90)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.90)
- Large Language Models are Good Prompt Learners for Low-Shot Image Classification (CVPR 2024 · similarity 0.89)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
