Research map / Label-efficient and robust learning / Semi-supervised learning
Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FineSSL, which fine-tunes a frozen CLIP ViT with VPT and adds balanced margin softmax and decoupled label smoothing to fix pseudo-label class bias and miscalibrated confidence in semi-supervised learning.
Problem
Semi-supervised learning with foundation models suffers from aggregated class biases in pseudo-labels and cognitive deviation (unreliable confidence), hurting pseudo-label selection in low-label regimes.
Approach
Adapt a frozen CLIP ViT via VPT and erase class bias directly in training using a balanced margin softmax with dynamic class margins from class learning pace, plus a decoupled auxiliary classifier trained with label smoothing that supplies sample reweighting weights without affecting the representation.
Key results
- FineSSL reaches 96.15% on CIFAR-10 N1 vs 78.30% for DebiasPL, the best baseline shown in the N1 column.
- On CIFAR-100 N4, FineSSL achieves 80.44% vs 79.57% for DebiasPL.
- Training time is about 1/6 of FixMatch with Wide ResNet trained from scratch (9630 s vs 55574.5 s on CIFAR-100).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FineSSL instead of the strongest listed baseline | CLIP-ViT SSL · CIFAR-10 N1 | accuracy: 78.30 ± 16.85 -> 96.15 ± 0.13 | Helped |
| Use FineSSL instead of the strongest listed baseline | CLIP-ViT SSL · CIFAR-10 N4 | accuracy: 97.43 ± 0.14 -> 97.39 ± 0.05 | No clear effect |
| Use FineSSL instead of the strongest listed baseline | CLIP-ViT SSL · CIFAR-100 N4 | accuracy: 79.57 ± 0.32 -> 80.44 ± 0.24 | Helped |
| Use FineSSL instead of the strongest listed baseline | CLIP-ViT SSL · FOOD-101 N2 | accuracy: 86.51 ± 0.74 -> 87.04 ± 0.43 | Helped |
| Use FineSSL instead of the strongest listed baseline | CLIP-ViT SSL · Semi-Aves with in- and out-of-distribution unlabeled data | Top-1 accuracy: 60.85 ± 0.37 -> 61.12 ± 0.07 | Helped |
| Use FineSSL instead of the strongest listed baseline | CLIP-ViT SSL · ImageNet 10% labeled | Top-1 accuracy: 78.77 -> 79.21 | Helped |
| Add balanced margin softmax (ablation) | FineSSL · CIFAR-100 N4 | accuracy: 76.70 -> 80.44 | Helped |
| Add margins to the supervised loss (ablation) | FineSSL · Semi-Aves with in- and out-of-distribution unlabeled data | accuracy: 58.94 -> 61.12 | Helped |
| Use dynamic rather than constant alpha (ablation) | FineSSL · CIFAR-100 N4 | accuracy: 80.30 -> 80.44 | Helped |
| Add decoupled label smoothing and its auxiliary classifier (ablation) | FineSSL · Semi-Aves with in- and out-of-distribution unlabeled data | accuracy: 58.58 -> 61.12 | Helped |
| Add label smoothing to the auxiliary classifier (ablation) | FineSSL · Semi-Aves with in- and out-of-distribution unlabeled data | accuracy: 59.49 -> 61.12 | Helped |
| Detach the auxiliary classifier from earlier layers (ablation) | FineSSL · CIFAR-100 N4 | accuracy: 79.85 -> 80.44 | Helped |
Surprising findings
- Plain FixMatch with VPT on CIFAR-100 shows overconfidence where N25 confidence exceeds N100, indicating confidence fails to track learning difficulty.
- Fine-tuning with full fine-tuning (FFT) performs far worse than LP or VPT in SSL unless LP-FT or semantic initialization is used.
Limitations
- Only the CLIP image encoder is used; text encoder is not exploited.
- Evaluated primarily on CLIP ViT; other self-supervised pretraining (SimCLR, MoCo) not studied.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 FOOD-101 Semi-Aves ImageNet
Benchmarks: CIFAR-10 CIFAR-100 FOOD-101 Semi-Aves ImageNet
Metrics: Top-1 accuracy training time pseudo-label accuracy and entropy
Baselines: Pseudo-Label FixMatch FlexMatch FreeMatch SoftMatch DebiasPL Supervised
Builds on: FixMatch FlexMatch visual prompt tuning DebiasPL SoftMatch
Closest papers by meaning
- Language-Assisted Debiasing and Smoothing for Foundation Model-Based Semi-Supervised Learning (CVPR 2025 · similarity 0.89)
- CGMatch: A Different Perspective of Semi-supervised Learning (CVPR 2025 · similarity 0.89)
- Revisiting Semi-Supervised Learning in the Era of Foundation Models (NeurIPS 2025 · similarity 0.89)
- A Unified Framework for Heterogeneous Semi-supervised Learning (CVPR 2025 · similarity 0.88)
- Enhancing Semi-Supervised Learning via Representative and Diverse Sample Selection (NeurIPS 2024 · similarity 0.87)
- Rethinking Confidence Scores and Thresholds in Pseudolabeling-based SSL (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.