AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?

Cheng-En Wu, Yu Tian, Haichao Yu, Heng Wang, Pedro Morgado, Yu Hen Hu, Linjie Yang

ICCV 2023 · Vision-language and multimodal › CLIP and zero-shot · approach: Prompt learning for CLIP · 25 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
keep CLIP text encoder fixed instead of fine-tuning it (ablation)CLIP prompt tuning · OxfordPets 16-shot, 50% label noiseaccuracy (%): 41.21 -> 73.13Improved
initialize classifier from CLIP text embeddings instead of random (ablation)linear probe · OxfordPets 16-shot, 50% label noiseaccuracy (%): 36.63 -> 44.21Improved
use prompt tuning through frozen text encoder instead of linear classifierCLIP · Food101 16-shot, 50% label noiseaccuracy (%): 34.37 -> 64.30Improved
make the classname token learnable (Full Prompt Tuning) (ablation)CLIP prompt tuning · OxfordPets 16-shot, 50% label noiseaccuracy (%): 73.13 -> 50.50Hurt
make classname token learnable at zero noise (ablation)CLIP prompt tuning · OxfordPets and Food101 16-shot, 0% noiseaccuracy (%): 87.89 -> 85.39 (OxfordPets)Hurt
use confusion noise instead of random noise at 50% rateprompt tuning · OxfordPets 16-shot, 50% label noiseaccuracy (%): 73.13 -> 66.55Hurt
use ViT-B/32 image encoder instead of RN50prompt tuning · ImageNet 16-shot, 50% label noiseaccuracy (%): 57.97 -> 61.75Improved
zero learnable context tokens (Ctx-0) under 50% noise (ablation)prompt tuning · context length studyaccuracyMixed
random sampling of pseudo-labels with GCE loss instead of top-confidence selectionUPL · ImageNet unsupervised prompt tuningaccuracy (%): 60.22 -> 62.14Improved
replace CE with GCE in Robust UPL (ablation)Robust UPL · DTD unsupervised prompt tuningaccuracy (%): 44.80 -> 46.69Improved
Robust UPL versus UPL on Caltech101Robust UPL · Caltech101 unsupervised prompt tuningaccuracy (%): 90.10 -> 88.07Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy