Research map / Label-efficient and robust learning / Semi-supervised learning
Evaluating multiple models using labeled and unlabeled data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSME, a semi-supervised mixture-model method that fits the joint distribution of classifier scores and labels from labeled plus unlabeled data (across multiple classifiers) to estimate metrics such as accuracy, ECE, AUC, and AUPRC.
Problem
Evaluating machine learning classifiers when labeled data is scarce, while unlabeled data is plentiful and multiple classifiers with probabilistic outputs are available.
Approach
Estimate the joint distribution P(y, s) of ground-truth labels and ALR-transformed multi-classifier scores with a semi-supervised mixture model fit by EM (KDE class-conditionals), then compute metrics using sampled labels from P(y|s) for unlabeled points.
Key results
- SSME reduces metric estimation error by 5.1x relative to labeled data alone and 2.4x relative to the next best method, averaged across tasks and metrics.
- With 20 labeled and 1000 unlabeled examples, SSME estimates accuracy within 1.5 percentage points on average, versus 3.4 for the closest baseline.
- SSME achieves an average effective sample size of 539 labeled examples for ECE versus 110 for the next best approach.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use labeled and unlabeled data with multiple classifiers and continuous scores instead of labeled data alone | SSME · Five binary tasks, averaged across tasks and metrics | metric estimation error: 5.1× relative to labeled data alone | Helped |
| Use labeled and unlabeled data with multiple classifiers and continuous scores instead of labeled data alone | SSME · Five binary tasks, accuracy | metric estimation error: 5.6× relative to using labeled data alone | Helped |
| Use labeled and unlabeled data with multiple classifiers and continuous scores instead of labeled data alone | SSME · Five binary tasks, AUC | metric estimation error: 2.9× relative to labeled data | Helped |
| Use labeled and unlabeled data with multiple classifiers and continuous scores instead of labeled data alone | SSME · Five binary tasks, ECE | metric estimation error: 7.2× reduction relative to labeled data alone | Helped |
| Use labeled and unlabeled data with multiple classifiers and continuous scores instead of labeled data alone | SSME · Five binary tasks, AUPRC | metric estimation error: 2.2× reduction relative to labeled data | Helped |
| Increase the labeled sample size from 20 to 50 or 100 examples | SSME · Five binary tasks, averaged performance estimation | error advantage vs labeled data: 5.6×, 3.0×, and 1.6× for 20, 50, and 100 datapoints | Mixed |
| Use labeled and unlabeled data with multiple classifiers and continuous scores instead of labeled data alone | SSME · MultiNLI and AG News, averaged across tasks and metrics | metric estimation error: 2.3× relative to labeled data | Helped |
| Use labeled and unlabeled data with multiple classifiers and continuous scores instead of labeled data alone | SSME · MIMIC-IV critical outcome, sex subgroup | metric estimation error: 5.3× relative to labeled data | Helped |
| Use labeled and unlabeled data with multiple classifiers and continuous scores instead of labeled data alone | SSME · MIMIC-IV critical outcome, race subgroup | metric estimation error: 2.6× relative to labeled data | Helped |
| Use labeled and unlabeled data with multiple classifiers and continuous scores instead of labeled data alone | SSME · MIMIC-IV critical outcome, age subgroup | metric estimation error: 2.4× relative to labeled data | Helped |
| Fit the mixture model to one classifier at a time rather than jointly to multiple classifiers (ablation) | SSME · Across metrics, tasks, and labeled-data amounts | performance estimation error | Hurt |
Limitations
- Assumes unlabeled data is drawn from the same distribution as labeled data.
- KDE-based density estimation can scale poorly in high-dimensional settings with many classifiers or classes.
- Theoretical analysis uses a stylized Gaussian mixture setting and a simplified UL+ estimator.
Experimental setup
Datasets: MIMIC-IV CivilComments OGB-SARS-CoV MultiNLI AG News
Benchmarks: MIMIC-IV CivilComments OGB-SARS-CoV MultiNLI AG News
Metrics: accuracy expected calibration error AUC AUPRC rescaled mean absolute error top-label calibration error
Baselines: Labeled Pseudo-Labeling Bayesian-Calibration AutoEval SPE Active-Testing Dawid-Skene Majority-Vote
Builds on: Dawid-Skene prediction-powered inference AutoEval Sheather-Jones bandwidth
Closest papers by meaning
- CGMatch: A Different Perspective of Semi-supervised Learning (CVPR 2025 · similarity 0.88)
- InterLUDE: Interactions between Labeled and Unlabeled Data to Enhance Semi-Supervised Learning (ICML 2024 · similarity 0.87)
- Scaling Up Semi-supervised Learning with Unconstrained Unlabelled Data (AAAI 2024 · similarity 0.86)
- Re-Evaluating the Impact of Unseen-Class Unlabeled Data on Semi-Supervised Learning Model (ICLR 2025 · similarity 0.86)
- Unlocking the Potential of Unlabeled Data in Semi-Supervised Domain Generalization (CVPR 2025 · similarity 0.86)
- Semi-supervised Multi-label Learning with Balanced Binary Angular Margin Loss (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.