Research map / Vision-language and multimodal / CLIP and zero-shot
Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Candidate Pseudolabel Learning (CPL), which fine-tunes CLIP with sets of candidate pseudolabels built from intra- and inter-instance confidence selection and trained with partial-label losses, iterating the candidate update during training.
Problem
Pseudolabel-based fine-tuning of vision-language models fails when zero-shot accuracy is low, because hard pseudolabels are often wrong and class-imbalanced.
Approach
Replace a single hard pseudolabel with a candidate set per instance, chosen by intra-instance cumulative-confidence thresholding (adaptive set size) intersected with inter-instance class-wise quantile selection over the confidence matrix, then train with existing partial-label losses and regenerate candidates iteratively.
Key results
- On EuroSAT with textual prompts under full unlabeled UL/SSL, CPL reaches 67.26% (UL) and 77.51% (SSL) versus GRIP's 57.21% and 58.66%, per the tables' SSL/UL columns.
- On DTD with visual prompts and ViT-B/32, CPL improves over GRIP by roughly 5% in SSL and UL settings (reported as 5.14% and 5.82% improvements in text for textual prompts).
- With LaFTer plus CPL in parameter-efficient UL, CPL reaches 76.7 on Flowers-102 and 82.2 on EuroSAT, best across six benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CPL instead of GRIP | CLIP with textual prompts and full unlabeled data · EuroSAT SSL | top-1 test accuracy (%): 58.66_{2.64} -> 77.51}_{0.80 | Helped |
| use CPL instead of GRIP | CLIP with textual prompts and full unlabeled data · EuroSAT UL | top-1 test accuracy (%): {57.21}_{1.77} -> 67.26}_{0.47 | Helped |
| use CPL instead of GRIP | CLIP with textual prompts and full unlabeled data · DTD SSL | top-1 test accuracy (%): {56.07}_{0.85} -> 61.21}_{0.56 | Helped |
| use CPL instead of GRIP | CLIP with textual prompts and full unlabeled data · Flowers102 SSL | top-1 test accuracy (%): {83.60}_{0.48} -> 89.66}_{0.36 | Helped |
| add inter-instance label selection (ablation) | CPL · CIFAR-100 balanced | top-1 test accuracy (%): 76.07 -> 77.32 | Helped |
| add inter-instance label selection (ablation) | CPL · CIFAR-100 imbalanced δ=100 | top-1 test accuracy (%): 66.68 -> 67.70 | Helped |
| add inter-instance label selection (ablation) | CPL · CIFAR-100 imbalanced δ=50 | top-1 test accuracy (%): 67.85 -> 69.65 | Helped |
| replace candidate-label CC loss with Soft CE (ablation) | CPL · DTD SSL (textual prompts, few-shot unlabeled data) | top-1 test accuracy (%): 54.63_{0.79} -> 51.83_{0.62} | Hurt |
| use normalized soft pseudolabel targets (ablation) | CPL · EuroSAT UL (textual prompts) | top-1 test accuracy (%): 67.26 -> 56.85 | Hurt |
| use normalized soft pseudolabel targets (ablation) | CPL · DTD UL (textual prompts) | top-1 test accuracy (%): 51.91 -> 49.53 | Hurt |
| vary λ from 0.50 to 1.50 (ablation) | CPL with textual prompts and full unlabeled data · Flowers102 SSL | top-1 test accuracy (%): 90.09; 90.28; 89.66; 89.68; 88.77 | Mixed |
| vary λ from 0.50 to 1.50 (ablation) | CPL with textual prompts and full unlabeled data · DTD SSL | top-1 test accuracy (%): 58.96; 60.13; 61.21; 59.31; 61.28 | Mixed |
Surprising findings
- Soft pseudolabels using normalized model confidences underperform treating all candidate labels equally, which the authors attribute to reinforcing confirmation bias.
- Average candidate set size stays close to 1 on most datasets, so the method rarely introduces high-entropy targets.
Limitations
- Performance depends on candidate quality; the true label may be absent from the candidate set.
- Hyperparameters alpha and beta are dataset-specific and require tuning; performance drops when beta is very large (fewer candidates, more unused samples).
- Class imbalance in unlabeled data still degrades performance somewhat (e.g., imbalanced CIFAR-100).
Experimental setup
Datasets: FGVC-Aircraft EuroSAT CUB Flowers102 RESISC45 DTD CALTECH-101 UCF-101 CIFAR-100
Benchmarks: Flowers102 RESISC45 DTD CUB EuroSAT FGVCAircraft CALTECH-101 UCF-101 CIFAR-100
Metrics: top-1 test accuracy harmonic mean of seen/unseen accuracy (TRZSL)
Baselines: Zero-shot CLIP FPL GRIP UPL CLIP-PR LaFTer Soft CE
Builds on: CLIP CoOp GRIP FPL LaFTer Classifier-Consistent loss
Closest papers by meaning
- Handling Imbalanced Pseudolabels for Vision-Language Models with Concept Alignment and Confusion-Aware Calibrated Margin (ICML 2025 · similarity 0.90)
- Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.89)
- Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model (ICLR 2025 · similarity 0.89)
- Active Prompt Learning in Vision Language Models (CVPR 2024 · similarity 0.88)
- Vision-Language Models are Strong Noisy Label Detectors (NeurIPS 2024 · similarity 0.88)
- Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.