Research map / Label-efficient and robust learning / Semi-supervised learning
Non-Stationary Predictions May Be More Informative: Exploring Pseudo-Labels with a Two-Phase Pattern of Training Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies "two-phase" predicted labels whose training dynamics switch class over epochs, selects them with a 2-phasic metric (LMO entropy plus temporal change measures), and adds them to existing pseudo-labeling methods with a tailored BCE-style loss, reporting gains on eight image and graph datasets.
Problem
Pseudo-label selection in semi-supervised learning usually relies on high confidence and high training stationarity, which discards predicted labels with non-stationary training dynamics that may carry useful information.
Approach
Instead of selecting stationary, high-confidence labels, select "two-phase" labels whose predicted class shifts from one category to another during training; a 2-phasic metric combines a bimodal spatial measure (Leave-Maximum-Out entropy over averaged softmax predictions) with a temporal measure of directional, small-magnitude prediction changes, and a loss pushes up the second-phase class and sup
Key results
- Adding two-phase labels improves baselines by an average of 1.73% on image datasets and 1.92% on graph datasets across eight datasets (Table 1).
- On Cora with GCN, two-phase labels reach 70.41% vs 66.21% for confidence at L/C=3 (Table 1).
- Two-phase labels have higher correctness than confidence selection at the top-k level on CIFAR-100 with 100 labels (61.83% vs 56.09%) and higher information gain, with low overlap (IoU 0.16-0.42) with confidence-selected labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 2-phase labels to pseudo-labeling methods | existing pseudo-labeling methods on image datasets · four image datasets | classification accuracy: 1.73% average gain | Helped |
| add 2-phase labels to pseudo-labeling methods | existing pseudo-labeling methods on graph datasets · four graph datasets | classification accuracy: 1.92% average gain | Helped |
| add two-phase pseudo-labels | Confidence pseudo-labeling · Cora (L/C=3) | accuracy: 66.21 -> 70.41 | Helped |
| add two-phase pseudo-labels | DR-GST pseudo-labeling · PubMed (L/C=3) | accuracy: 68.88 -> 73.06 | Helped |
| add two-phase pseudo-labels | MoDis pseudo-labeling · CIFAR100 (# Labels=100) | accuracy: 54.06 -> 58.40 | Helped |
| add two-phase pseudo-labels | SoftMatch pseudo-labeling · EuroSAT (# Labels=20) | accuracy: 91.30 -> 95.20 | Helped |
| select pseudo-labels using the 2-phasic metric rather than confidence | pseudo-label selection · CIFAR100 (# Labels=100) | correctness of pseudo-labels: 56.09 -> 61.83 | Helped |
| select pseudo-labels using the 2-phasic metric rather than confidence | pseudo-label selection · Cora (L/C=3) | information gain of pseudo-labels: 2.81 -> 3.78 | Helped |
| include the c1 loss term in addition to c2 (ablation) | two-phase pseudo-labeling loss · Cora and CIFAR100 | accuracy: Cora c1 improvement: 1.20, 1.16, 0.11; CIFAR100 c1 improvement: 1.07, 0.17, 0.21 | Helped |
| use only the temporal measure rather than the full 2-phasic metric (ablation) | 2-phasic pseudo-label selection · Cora | accuracy: 69.76, 73.40, 75.99; full 2-phasic: 70.16, 74.07, 77.17 | Hurt |
| use only the spatial measure rather than the full 2-phasic metric (ablation) | 2-phasic pseudo-label selection · Cora | accuracy: 68.24, 72.70, 73.87; full 2-phasic: 70.16, 74.07, 77.17 | Hurt |
Surprising findings
- Labels that are non-stationary and have relatively high confidence can be more informative for pseudo-labeling than stationary high-confidence labels.
- Two-phase labels are largely disjoint from confidence-selected labels (IoU as low as 0.16), so they are overlooked by confidence-based selection.
Limitations
- Hyperparameters (eta, phi, epsilon, tau) must be manually or Optuna-tuned and optimal values differ across datasets.
- Gains are reported on small labeled-data settings; generalization to larger-scale or other learning scenarios is untested.
- Requires recording training dynamics over multiple epochs, adding memory overhead proportional to unlabeled set size.
Experimental setup
Datasets: CIFAR-100 EuroSAT Semi-Aves STL-10 Cora Citeseer PubMed AmazonComputers
Benchmarks: CIFAR-100 EuroSAT Semi-Aves STL-10 Cora Citeseer PubMed AmazonComputers
Metrics: classification accuracy pseudo-label correctness information gain IoU overlap
Baselines: Confidence score AUM MoDis DR-GST UPS SoftMatch
Builds on: MoDis Softmatch UPS AUM DR-GST
Closest papers by meaning
- Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning (AAAI 2021 · similarity 0.88)
- Semi-Supervised Regression with Heteroscedastic Pseudo-Labels (NeurIPS 2025 · similarity 0.87)
- Bi-Level Optimization for Semi-Supervised Learning with Pseudo-Labeling (AAAI 2025 · similarity 0.87)
- Keep It on a Leash: Controllable Pseudo-label Generation Towards Realistic Long-Tailed Semi-Supervised Learning (NeurIPS 2025 · similarity 0.87)
- Leveraging Local Variance for Pseudo-Label Selection in Semi-supervised Learning (AAAI 2024 · similarity 0.86)
- Dynamic Pseudo Labeling via Gradient Cutting for High-Low Entropy Exploration (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.