Research map / Label-efficient and robust learning / Recognition and long-tail
Confidence calibration objectives
Calibrates network confidence with soft objectives, pairwise constraints and ensemble methods.
129 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Recognition and long-tail, keyword cluster “label · classification · classifier”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 2 of 5 tracked benchmarks.
44 papers in 2023–24 and 41 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 3.5% to 3.1%.
Open questions the data points to
- Tiny-ImageNet (ECE): best went from 1.99 (up to 2024) to 1.35 in 2025 — Balancing Two Classifiers via A Simplex ETF Structure for Model Calibration (CVPR 2025 · 1 citation)
- Tiny-ImageNet (ACC): best went from 50.8 (up to 2024) to 67.6 in 2025 — Tailoring Mixup to Data for Calibration (ICLR 2025 · 0 citations)
What looks settled
- CIFAR-10 (accuracy): best up to 2024 96.1, best in 2025–26 96.42 from 4 papers, less than 1% better.
- CIFAR-100 (accuracy): best up to 2024 79.47, best in 2025–26 79.84 from 4 papers, less than 1% better.
- SVHN (accuracy): best up to 2024 96.81, best in 2025–26 96.79 from 2 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples (ICLR 2018 · 113 citations)
- 2019 · Why ReLU Networks Yield High-Confidence Predictions Far Away From the Training Data and How to Mitigate the Problem (CVPR 2019 · 443 citations)
- 2020 · Uncertainty-Aware Score Distribution Learning for Action Quality Assessment (CVPR 2020 · 197 citations)
- 2021 · Post-Hoc Uncertainty Calibration for Domain Drift Scenarios (CVPR 2021 · 55 citations)
- 2022 · The Devil Is in the Margin: Margin-Based Label Smoothing for Network Calibration (CVPR 2022 · 70 citations)
- 2023 · Deep Deterministic Uncertainty: A New Simple Baseline (CVPR 2023 · 102 citations)
- 2024 · Towards Calibrated Multi-label Deep Neural Networks (CVPR 2024 · 18 citations)
- 2025 · Risk Controlled Image Retrieval (AAAI 2025 · 3 citations)
- 2026 · Minimum-Length Conformal Prediction Sets for Ordinal Classification (AAAI 2026 · 1 citation)
Earliest papers here
- Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples (ICLR 2018 · 113 citations)
- Why ReLU Networks Yield High-Confidence Predictions Far Away From the Training Data and How to Mitigate the Problem (CVPR 2019 · 443 citations)
- On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks (NeurIPS 2019 · 428 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
CIFAR-10 · accuracy (higher is better)
7 papers, 2020–2025 · small gains only
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 96.02 | Auxiliary Training: Towards Accurate and Robust Models (CVPR 2020 · 35 citations) |
| 2023 | 96.1 | On the Pitfall of Mixup for Uncertainty Calibration (CVPR 2023 · 10 citations) |
| 2025 | 96.42 | Tailoring Mixup to Data for Calibration (ICLR 2025 · 0 citations) |
CIFAR-100 · accuracy (higher is better)
7 papers, 2020–2025 · small gains only
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 79.47 | Auxiliary Training: Towards Accurate and Robust Models (CVPR 2020 · 35 citations) |
| 2025 | 79.84 | Training Deep Neural Networks with Virtual Smoothing Classes (AAAI 2025) |
Tiny-ImageNet · ECE (lower is better)
5 papers, 2020–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 3.6 | Distance-Based Learning from Errors for Confidence Calibration (ICLR 2020 · 14 citations) |
| 2022 | 1.99 | A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibration (CVPR 2022 · 42 citations) |
| 2025 | 1.35 | Balancing Two Classifiers via A Simplex ETF Structure for Model Calibration (CVPR 2025 · 1 citation) |
SVHN · accuracy (higher is better)
4 papers, 2020–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 96.81 | Confidence-Aware Learning for Deep Neural Networks (ICML 2020 · 45 citations) |
Tiny-ImageNet · ACC (higher is better)
4 papers, 2023–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 50.8 | On the Pitfall of Mixup for Uncertainty Calibration (CVPR 2023 · 10 citations) |
| 2025 | 67.6 | Tailoring Mixup to Data for Calibration (ICLR 2025 · 0 citations) |
What has been tried
101 papers here report 1,087 measured changes (a component added, removed or swapped, compared with a baseline): helped 66%, no clear effect 8%, hurt 21%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add LPR to SPA | MS-COCO · ACE | 5.3 -> 4.2 | Helped | Towards Calibrated Multi-label Deep Neural Networks (CVPR 2024 · 18 citations) |
| Add neighboring labeling | EyeDiap · angle error (°) | 5.16 -> 5.05 | Helped | Suppressing Uncertainty in Gaze Estimation (AAAI 2024 · 11 citations) |
| add high- and low-reliable pseudo-label losses | CIFAR-100 W4A4 · Top-1 accuracy | 66.32 -> 66.94 | Helped | Data-Free Quantization via Pseudo-label Filtering (CVPR 2024 · 5 citations) |
| Add SRTM slope filtering to exclude unreliable GEDI labels | Needle Mountains ALS · MAE (m) | 9.77 -> 7.33 | Helped | Estimating Canopy Height at Scale (ICML 2024 · 4 citations) |
| use RLLS importance weights rather than ELSA weights | CIFAR-100-LT (IF=10), ResNet-20 · estimated classwise CE ×100 | 96.61±0.54 -> 63.97±0.28 | Helped | LaSCal: Label-Shift Calibration without target labels (NeurIPS 2024 · 4 citations) |
| Replace MCD's uncertainty estimator with a random uncertainty estimator | Risk-control experiment · empirical risk | No clear effect | Risk Controlled Image Retrieval (AAAI 2025 · 3 citations) | |
| Add survival-outcome-weighted contrastive loss (SNCE) to NLL | METABRIC, NWTCO, GBSG, FLCHAIN, SUPPORT, SEER · CI | 0.608 (±0.046) -> 0.688 (±0.017); 0.711 (±0.051) -> 0.729 (±0.052); 0.675 (±0.018) -> 0.696 (±0.006); 0.794_ | Helped | Toward a Well-Calibrated Discrimination via Survival Outcome-Aware Contrastive Learning (NeurIPS 2024 · 3 citations) |
| Apply TvA to Histogram Binning | C100 · ECE (%) | 9.56 -> 1.02 | Helped | Confidence Calibration of Classifiers with Many Classes (NeurIPS 2024 · 3 citations) |
| add temperature scaling | ImageNet-1k validation · hAURC (× 1000) | 39.24±0.49 -> 36.51±0.47 | Helped | Hierarchical Selective Classification (NeurIPS 2024 · 3 citations) |
| add in-training uncertainty | LibriSpeech test-clean · WER | 4.36 -> 4.31 | Helped | Uncertainty-Aware Self-Training for CTC-Based Automatic Speech Recognition (AAAI 2025 · 2 citations) |
| use differentiated learning rates and schedules | CIFAR-100 · ECE (%) | 4.42 -> 1.38 | Helped | Improving Accuracy and Calibration via Differentiated Deep Mutual Learning (CVPR 2025 · 2 citations) |
| add distance-based softened labels to vicinal samples | DUTS-TE · ECE | 3.05 -> 0.78 | Helped | Self-Calibrating Vicinal Risk Minimisation for Model Calibration (CVPR 2024 · 2 citations) |
What reviewers will expect
Datasets and baselines named in the 24 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 12 of 24 papers
- CIFAR-100 · 10 of 24 papers
- Tiny-ImageNet · 6 of 24 papers
- ImageNet · 5 of 24 papers
- SVHN · 4 of 24 papers
- MNIST · 2 of 24 papers
- ImageNet10 · 2 of 24 papers
Baselines
- Temperature Scaling · 5 of 24 papers
- Vector Scaling · 4 of 24 papers
- label smoothing · 3 of 24 papers
- Mixup · 3 of 24 papers
- MMCE · 3 of 24 papers
- Histogram Binning · 2 of 24 papers
- FLSD · 2 of 24 papers
- CPC · 2 of 24 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 85 papers; from OpenAlex and the papers' affiliations).
- The Chinese University of Hong Kong · 4 papers
- Zhejiang Lab · 3 papers
- Tsinghua University · 3 papers
- Technion - Israel Institute of Technology · 3 papers
- University of Pennsylvania · 3 papers
- Microsoft · 3 papers
- Washington State University · 2 papers
- Northeastern University · 2 papers
- KU Leuven · 2 papers
- Deci · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Partial and multi-label learning (345 papers)
- Loss design and classifier theory (273 papers)
- Group-robust classification (251 papers)
- Learning from unlabeled and positive-unlabeled data (240 papers)
- Deep active learning (76 papers)
Elsewhere in the topic
- Multilingual language representation pretraining (507 papers)
- Contrastive representation theory (495 papers)
- Dense features and foundation priors (482 papers)
- Correspondence and causal representation learning (447 papers)
- Data augmentation and generalization bias (409 papers)
- Recommendation representation learning (207 papers)
- Margin-based face recognition losses (124 papers)
- Supervised person re-identification (108 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).