Research map / Label-efficient and robust learning / Noisy labels
Pre-Trained Vision-Language Models as Noisy Partial Annotators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses CLIP with 27 prompt templates to annotate unlabeled images as noisy partial labels, then trains two PreAct ResNet-18 networks with a collaborative consistency regularization NPLL method (Co-Reg) that outperforms zero-shot, KD, other weakly supervised methods, and CoOp on several image classification benchmarks.
Problem
Adapting pre-trained vision-language models like CLIP to downstream image classification without human annotation, when zero-shot performance is poor and deployment models must be small; CLIP-generated labels are noisy and form partial label sets.
Approach
Treat each prompt template's top-1 prediction from CLIP as a candidate label, forming noisy partial label sets, and train two networks that co-purify labels via a GMM-based small-loss split, co-pseudo-label, self-train, and refine representations with prototypical alignment and noisy supervised contrastive learning.
Key results
- Co-Reg reaches 94.06% on CIFAR-10 and 71.04% on CIFAR-100 versus 88.51% and 61.55% for CLIP zero-shot on test sets, beating all P-WSL baselines on 6 of 7 datasets.
- On SVHN, Co-Reg achieves 46.57% against 8.63% zero-shot, and 65.54% on EuroSAT versus 31.49% zero-shot.
- Co-Reg outperforms CoOp at 2/4 shots on five of seven datasets; ablation shows removing co-pseudo-labeling drops CIFAR-100 accuracy from 71.04% to 68.40%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Co-Reg instead of CLIP zero-shot inference | CLIP ViT-B/32 on CIFAR-10 · CIFAR-10 | accuracy: 88.51% -> 94.06% | Helped |
| Use Co-Reg instead of CLIP zero-shot inference | CLIP ViT-B/32 on CIFAR-100 · CIFAR-100 | accuracy: 61.55% -> 71.04% | Helped |
| Use Co-Reg instead of CLIP zero-shot inference | CLIP ViT-B/32 on SVHN · SVHN | accuracy: 8.63% -> 46.57% | Helped |
| Use Co-Reg instead of CLIP zero-shot inference | CLIP ViT-B/32 on FMNIST · FMNIST | accuracy: 61.46% -> 76.28% | Helped |
| Use Co-Reg instead of CLIP zero-shot inference | CLIP ViT-B/32 on EuroSAT · EuroSAT | accuracy: 31.49% -> 65.54% | Helped |
| Use Co-Reg instead of CLIP zero-shot inference | CLIP ViT-B/32 on FER2013 · FER2013 | accuracy: 42.07% -> 50.26% | Helped |
| Use Co-Reg instead of CLIP zero-shot inference | CLIP ViT-B/32 on GTSRB · GTSRB | accuracy: 25.14% -> 41.18% | Helped |
| Replace Co-Reg with CoOp few-shot fine-tuning | CLIP ViT-B/32 on the seven image classification datasets · CIFAR-10, CIFAR-100, SVHN, FMNIST, EuroSAT, FER2013, GTSRB | accuracy | Mixed |
| Add collaborative pseudo-labeling (ablation) | Co-Reg on CIFAR-100 · CIFAR-100 | accuracy: 68.40% -> 71.04% | Helped |
| Add noisy supervised contrastive learning (ablation) | Co-Reg on CIFAR-100 · CIFAR-100 | accuracy: 69.33% -> 71.04% | Helped |
| Replace traditional supervised contrastive learning with noisy supervised contrastive learning (ablation) | Co-Reg on CIFAR-100 · CIFAR-100 | accuracy: 67.96% -> 71.04% | Helped |
| Add prototypical similarity alignment (ablation) | Co-Reg on CIFAR-100 · CIFAR-100 | accuracy: 69.81% -> 71.04% | Helped |
Surprising findings
- Directly applying traditional supervised contrastive learning on pseudo-labeled data lowers CIFAR-100 accuracy (67.96%) relative to the full method, due to pseudo-label noise.
- CLIP predicts almost all SVHN images as 'number 0' with zero-shot accuracy under 10%, yet the partial-label candidate sets still allow effective training.
Limitations
- Requires a large number of unlabeled downstream samples (about 200 per class on CIFAR-100 to beat zero-shot).
- Method is not designed for synthetic NPLL settings, and on some datasets (e.g., FER2013, GTSRB) it is not best; ALIM variants win on some tasks.
- Experiments limited to image classification with CLIP ViT-B/32 and a small ResNet-18 backbone.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 SVHN Fashion-MNIST EuroSAT FER2013 GTSRB
Benchmarks: CIFAR-10 CIFAR-100 SVHN Fashion-MNIST EuroSAT FER2013 GTSRB
Metrics: accuracy
Baselines: Zero-Shot CLIP Knowledge distillation (unsupervised) CR-DPLL DivideMix ALIM-Onehot ALIM-Scale CoOp CC RC LWC LWS PiCO
Builds on: DivideMix MoCo PiCO CoOp ALIM
Closest papers by meaning
- Progressive Pseudo-label Self-balancing Towards Unsupervised Vision-Language Models Adaptation (NeurIPS 2026 · similarity 0.89)
- Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations (CVPR 2024 · similarity 0.87)
- Controller-Guided Partial Label Consistency Regularization with Unlabeled Data (AAAI 2024 · similarity 0.87)
- Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Use (CVPR 2024 · similarity 0.86)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.86)
- Generative Modeling Reinvents Supervised Learning: Label Repurposing with Predictive Consistency Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.