Research map / Label-efficient and robust learning / Semi-supervised learning
Bi-Level Optimization for Semi-Supervised Learning with Pseudo-Labeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BOPL, a bi-level optimization method that treats pseudo-labels as latent variables and optimizes them directly via a validation loss on an inner-trained student model, plus an ICT-based fine-tuning variant.
Problem
Pseudo-labeling in semi-supervised learning often produces low-quality pseudo-labels and lacks a principled way to keep pseudo-labels accurate and consistent with labeled predictions.
Approach
Formulate pseudo-labeling as a bi-level problem: inner loop trains the student on soft pseudo-labels, outer loop updates the pseudo-labels to minimize a validation loss (labeled cross-entropy plus unlabeled entropy) on the resulting model, with a one-step approximation and finite-difference gradient; then fine-tune with MSE or ICT consistency.
Key results
- BOPL+ICT reaches 4.12% test error on CIFAR-10 with 250 labels (WRN-28-2), versus 4.84-5.07% for other listed methods shown.
- BOPL achieves 4.93% and BOPL+ICT 4.11% error on STL-10 with 1000 labels (WRN-37-2), beating FixMatch (5.17%) and MarginMatch (5.52%).
- Ablation on CIFAR-100 (CNN-13, 4000 labels): removing labeled outer loss raises error from 36.78% to 45.34%, the largest degradation among variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ICT to BOPL fine-tuning | BOPL · CIFAR-10, 250 labeled | test error: 4.65 -> 4.12 | Helped |
| Add ICT to BOPL fine-tuning | BOPL · CIFAR-100, 2,500 labeled | test error: 24.84 -> 23.16 | Helped |
| Add ICT to BOPL fine-tuning | BOPL · SVHN, 1,000 labeled | test error: 1.81 -> 1.78 | Helped |
| Add ICT to BOPL fine-tuning | BOPL · STL-10, 1,000 labeled | test error: 4.93 -> 4.11 | Helped |
| Add teacher EMA updates (ablation) | BOPL · CIFAR-100, 4,000 labeled | test error: 40.01(0.54) -> 36.78(0.29) | Helped |
| Add teacher EMA updates (ablation) | BOPL · CIFAR-100, 10,000 labeled | test error: 31.75(0.43) -> 28.92(0.18) | Helped |
| Add unlabeled-data outer loss (ablation) | BOPL · CIFAR-100, 4,000 labeled | test error: 38.76(0.49) -> 36.78(0.29) | Helped |
| Add unlabeled-data outer loss (ablation) | BOPL · CIFAR-100, 10,000 labeled | test error: 30.12(0.42) -> 28.92(0.18) | Helped |
| Add labeled-data outer loss (ablation) | BOPL · CIFAR-100, 4,000 labeled | test error: 45.34(0.63) -> 36.78(0.29) | Helped |
| Add fine-tuning (ablation) | BOPL · CIFAR-100, 10,000 labeled | test error: 35.80(0.45) -> 28.92(0.18) | Helped |
| Add Mean Teacher pre-training (ablation) | BOPL · CIFAR-100, 4,000 labeled | test error: 44.15(0.57) -> 36.78(0.29) | Helped |
| Integrate teacher predictions in the ReLU pseudo-label update (ablation) | BOPL pseudo-label update · CIFAR-100, 4,000 labeled | test error: 37.11(0.38) -> 36.78(0.29) | Helped |
Surprising findings
- Using ReLU-based normalization of pseudo-label updates outperformed softmax normalization in the ablation.
Limitations
- Bi-level optimization adds training cost from the finite-difference gradient approximation and pre-training stages.
- Evaluated only on image classification benchmarks with standard CNN/WRN backbones.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 SVHN STL-10
Benchmarks: CIFAR-10 CIFAR-100 SVHN STL-10
Metrics: mean test error standard deviation over 5 runs
Baselines: Mean Teacher MixMatch UDA ReMixMatch FixMatch FlexMatch CoMatch SimMatch Meta-Semi Meta Pseudo-Labels ICT MarginMatch
Builds on: Mean Teacher Interpolation Consistency Training Meta Pseudo-Labels FixMatch
Closest papers by meaning
- Semi-Supervised Regression with Heteroscedastic Pseudo-Labels (NeurIPS 2025 · similarity 0.90)
- Keep It on a Leash: Controllable Pseudo-label Generation Towards Realistic Long-Tailed Semi-Supervised Learning (NeurIPS 2025 · similarity 0.90)
- Rethinking Confidence Scores and Thresholds in Pseudolabeling-based SSL (ICML 2025 · similarity 0.89)
- SemiReward: A General Reward Model for Semi-supervised Learning (ICLR 2024 · similarity 0.89)
- Leveraging Local Variance for Pseudo-Label Selection in Semi-supervised Learning (AAAI 2024 · similarity 0.88)
- CGMatch: A Different Perspective of Semi-supervised Learning (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.