Research map / Label-efficient and robust learning / Recognition and long-tail
Pearls from Pebbles: Improved Confidence Functions for Auto-labeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Colander, a post-hoc learned confidence function for threshold-based auto-labeling that directly optimizes coverage under an error constraint via a differentiable surrogate, outperforming calibration baselines.
Problem
Threshold-based auto-labeling (TBAL) needs confidence scores that separate correct from incorrect predictions so that many points can be auto-labeled under an error tolerance; softmax and calibration methods produce overconfident scores and low coverage.
Approach
Frame the optimal TBAL confidence function as a coverage-maximization problem under an error constraint, relax the 0-1 indicators with sigmoid surrogates, and learn a two-layer network on the model's last two representations as the confidence function, with thresholds estimated on held-out validation data.
Key results
- Colander reaches up to about 60% coverage gain over baselines on Tiny-ImageNet (61.2% vs 37.4% for best prior post-hoc) while keeping error below 5%.
- On CIFAR-10 with Vanilla training, Colander achieves 78.5% coverage at 3.0% error vs 14.0% coverage for softmax at 4.8% error.
- Train-time ordinal ranking methods CRL and FMFP showed no significant gain over Vanilla in TBAL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace softmax confidence scores with Colander | Vanilla TBAL · CIFAR-10 | coverage: 14.0 ± 2.1 -> 78.5 ± 0.2 | Helped |
| replace softmax confidence scores with Colander | Vanilla TBAL · MNIST | coverage: 85.0 ± 2.5 -> 95.6 ± 1.4 | Helped |
| replace softmax confidence scores with Colander | Vanilla TBAL · 20 Newsgroups | coverage: 48.2 ± 1.6 -> 80.6 ± 0.7 | Helped |
| replace softmax confidence scores with Colander | Vanilla TBAL · Tiny-ImageNet | coverage: 32.6 ± 0.5 -> 59.2 ± 0.8 | Helped |
| use pre-logits instead of concatenated features (ablation) | Colander · 20 Newsgroups | coverage: 82.9 ± 0.4 -> 82.8 ± 0.5 | No clear effect |
| use logits instead of concatenated features (ablation) | Colander · 20 Newsgroups | coverage: 82.9 ± 0.4 -> 82.8 ± 0.3 | No clear effect |
| use pre-logits instead of concatenated features (ablation) | Colander · CIFAR-10 | coverage: 79.0 ± 0.3 -> 79.0 ± 0.2 | No clear effect |
| use logits instead of concatenated features (ablation) | Colander · CIFAR-10 | coverage: 79.0 ± 0.3 -> 76.5 ± 0.9 | Hurt |
Surprising findings
- Train-time methods designed for ordinal ranking (CRL, FMFP) do not improve TBAL over vanilla training, attributed to limited training data and zero training error.
- Squentropy with softmax gives unexpectedly large coverage gains on Tiny-ImageNet (6-7%).
Limitations
- Like other post-hoc methods, requires held-out validation data to learn the confidence function.
- Hyperparameters selected only in the first TBAL round via grid search, which may be suboptimal in later rounds.
- Evaluated only on four datasets with small models (SimpleCNN, LeNet-5, MLPs on embeddings).
Experimental setup
Datasets: MNIST CIFAR-10 Tiny-ImageNet 20 Newsgroups
Benchmarks: MNIST CIFAR-10 Tiny-ImageNet 20 Newsgroups
Metrics: auto-labeling error coverage
Baselines: Softmax Temperature scaling Scaling-binning Dirichlet calibration Top-label histogram binning CRL FMFP Squentropy
Builds on: temperature scaling Dirichlet calibration correctness ranking loss FMFP Squentropy
Closest papers by meaning
- Promises and Pitfalls of Threshold-based Auto-labeling (NeurIPS 2023 · similarity 0.93)
- Provable Selective Auto-labeling with Reliability Guarantees (NeurIPS 2026 · similarity 0.90)
- Enhanced Sample Selection with Confidence Tracking: Identifying Correctly Labeled Yet Hard-to-Learn Samples in Noisy Data (AAAI 2025 · similarity 0.86)
- Learning from Noisy Labels via Conditional Distributionally Robust Optimization (NeurIPS 2024 · similarity 0.86)
- Rethinking Confidence Scores and Thresholds in Pseudolabeling-based SSL (ICML 2025 · similarity 0.85)
- Conformal Prediction for Partial Label Learning (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.