Research map / Vision-language and multimodal / CLIP and zero-shot
Vision-Language Models are Strong Noisy Label Detectors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeFT, a two-phase framework that uses CLIP dual (positive/negative) learnable textual prompts with VPT-adapted visual encoder as a noisy label detector, then full fine-tunes on selected clean samples.
Problem
Fine-tuning vision-language models like CLIP on downstream datasets with noisy labels degrades performance, and existing learning-with-noisy-labels methods were developed for training from scratch and ignore multimodal pre-trained knowledge.
Approach
Learn a positive and a negative textual prompt per class; the negative prompt's similarity acts as a learnable, sample-dependent threshold separating clean from noisy samples. Visual encoder is adapted with PEFT (VPT) to align with the prompts, and the detected clean subset is then used for full fine-tuning with a linear classifier.
Key results
- On Tiny-ImageNet with 60% symmetric noise, DeFT improves noisy-label detection precision and recall over small-loss by 4.58% and 4.55%; on CUB-200-2011 the gains are 9.78% and 14.42%.
- On CIFAR-100N, DeFT reaches 79.04% test accuracy versus 76.06% for GMM and 77.68% for UNICON; on Clothing1M it reaches 72.44%.
- On Stanford-Cars, DeFT boosts test accuracy by an average of 4.34% over the best comparison method across synthetic noise settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DeFT instead of the strongest listed baseline | CLIP fine-tuning · Clothing1M | test accuracy: 72.14 -> 72.44 | Helped |
| Use DeFT instead of the strongest listed baseline | CLIP fine-tuning · WebVision | test accuracy: 84.96 -> 85.12 | Helped |
| Use DeFT instead of Small-loss | noisy label detection · CIFAR-100, Sym. 0.6 | precision: 92.93 -> 97.04 | Helped |
| Use DeFT instead of Small-loss | noisy label detection · CIFAR-100, Sym. 0.6 | recall: 90.68 -> 97.27 | Helped |
| Use DeFT instead of Small-loss | noisy label detection · CIFAR-100, Sym. 0.8 | F1-score: 86.87 -> 93.39 | Helped |
| Replace PEFT with FFT for model adaptation on selected clean samples (ablation) | DeFT · CIFAR-100, Sym. 0.4 | test accuracy (best / last): 86.69 / 86.69 -> 88.17 / 88.11 | Helped |
| Replace FFT with PEFT for model adaptation on selected clean samples (ablation) | DeFT · CIFAR-100, Sym. 0.4 | test accuracy (best / last): 88.17 / 88.11 -> 86.69 / 86.69 | Hurt |
| Add the model adaptation phase using FFT on selected clean samples (ablation) | DeFT · Stanford-Cars and CUB-200-2011 | test accuracy | Helped |
| Increase visual prompt length across tested values (ablation) | DeFT noisy label detector · CIFAR-100, Sym. 0.4 | F1-score: 98.08; 98.17; 98.33; 98.30 | No clear effect |
| Increase textual prompt length from 4 to 8 (ablation) | DeFT noisy label detector · CIFAR-100, Sym. 0.4 | F1-score: 93.29 -> 97.14 | Helped |
| Increase textual prompt length from 8 to 16 or 32 (ablation) | DeFT noisy label detector · CIFAR-100, Sym. 0.4 | F1-score: 97.14; 97.15; 97.31 | Helped |
| Vary training epochs from 5 to 20 (ablation) | DeFT noisy label detector · CIFAR-100, Sym. 0.4 | F1-score: 96.65; 97.15; 97.34; 97.38 | No clear effect |
Surprising findings
- Full fine-tuning distorts pre-trained representations under heavy label noise, so PEFT (VPT) is more robust than FFT for the detection phase, while FFT is better for the final adaptation on clean data.
- Textual-prompt classification (VLPT) is more robust to noise than a linear probe on VPT features, yet VLPT performs worst on clean datasets.
Limitations
- Only handles single-label multi-class classification with noisy image-label pairs; not noisy image-text pairs or multi-label settings.
- Relies on a warm-up stage and, for real-world datasets, an added consistency constraint with predicted labels; high-noise settings need tuned learning rates and loss weights.
Experimental setup
Datasets: CIFAR-100 Tiny-ImageNet Stanford-Cars CUB-200-2011 CIFAR-100N Clothing1M WebVision
Benchmarks: CIFAR-100 Tiny-ImageNet Stanford-Cars CUB-200-2011 CIFAR-100N Clothing1M WebVision
Metrics: precision recall F1-score test accuracy (best/last)
Baselines: Label-match Small-loss CE SCE ELR GMM RoLT UNICON LongReMix ProMix GCE TURN
Builds on: CLIP CoOp Visual Prompt Tuning DivideMix negative learning
Closest papers by meaning
- Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning (ICML 2024 · similarity 0.88)
- Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models (ICML 2025 · similarity 0.88)
- Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data (ICML 2024 · similarity 0.88)
- NLPrompt: Noise-Label Prompt Learning for Vision-Language Models (CVPR 2025 · similarity 0.88)
- Enhance Vision-Language Alignment with Noise (AAAI 2025 · similarity 0.88)
- VladVA: Discriminative Fine-tuning of LVLMs (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.