Research map / Label-efficient and robust learning / Recognition and long-tail

Confidence Calibration of Classifiers with Many Classes

NeurIPS 2024 · Label-efficient and robust learning › Recognition and long-tail · approach: Confidence calibration objectives · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Reformulates multiclass confidence calibration as a single surrogate binary "is the prediction correct?" problem (Top-versus-All), letting existing scaling and binary calibrators work better with many classes.

Problem

Post-hoc confidence calibration of multiclass neural classifiers degrades for problems with many classes: One-versus-All binary calibration is highly imbalanced, scaling methods like Vector Scaling and Dirichlet Calibration overfit, and per-class calibrators can change the predicted class.

Approach

Replace the multiclass calibration with one binary surrogate classifier whose score is the max class probability and whose label is prediction correctness; calibrate it with standard methods (binary cross-entropy for scaling methods, plus L2 regularization on vector coefficients) and apply the learned mapping to confidences only, leaving predictions unchanged.

Key results

• On ImageNet, Histogram Binning with TvA reduces ECE for ResNet-50 from 7.49% (HB) to 0.55%, and the TvA variants of binary methods are mostly below 1% ECE.
• Vector Scaling with TvA and L2 regularization (VS_reg_TvA) reduces ImageNet-21K MN3 ECE from 2.52% (VS) to 2.40%, and ViT-B/16 from 2.38% to 1.54%, and keeps improving with more calibration data, outperforming TS_TvA beyond about 15000 samples.
• TvA reduces ECE for most scaling and binary methods across image and text datasets, with larger gains on datasets with more classes; HB_TvA gives the lowest ECE in most settings.

Measured effects reported in the paper

Surprising findings

• ImageNet networks are mostly underconfident after calibration-free evaluation, contrary to the common belief that modern networks are overconfident.
• CLIP has the lowest uncalibrated ECE on ImageNet but the highest ECE after several TvA-based calibrations, which the authors attribute to its training regime.

Limitations

• Confidence calibration only; does not address stronger notions such as class-wise calibration.
• Improvements are smaller for problems with few classes (10 or fewer).
• Regularization adds a hyperparameter lambda for Vector Scaling and Dirichlet Calibration, and DC is sensitive to hyperparameter tuning and sometimes diverges.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 ImageNet ImageNet-21K Amazon Fine Foods DynaSent MNLI Yahoo Answers

Benchmarks: CIFAR-10 CIFAR-100 ImageNet ImageNet-21K Amazon Fine Foods DynaSent MNLI Yahoo Answers

Metrics: ECE Brier score AUROC MCE ACE PIECE accuracy

Baselines: Temperature Scaling Vector Scaling Dirichlet Calibration Isotonic Regression Beta Calibration Bayesian Binning into Quantiles Histogram Binning I-Max IRM ProCal Multi Isotonic Regression

Builds on: Temperature Scaling Vector Scaling Dirichlet Calibration Histogram Binning Isotonic Regression Beta Calibration Bayesian Binning into Quantiles I-Max IRM

Closest papers by meaning

• On Calibration of Modern Neural Networks (ICML 2017 · similarity 0.89)
• Improving Multi-Class Calibration through Normalization-Aware Isotonic Techniques (ICML 2025 · similarity 0.86)
• Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling (AAAI 2025 · similarity 0.86)
• Truthful Calibration Errors for Multi-Class Prediction (NeurIPS 2026 · similarity 0.85)
• Tilt and Average : Geometric Adjustment of the Last Layer for Recalibration (ICML 2024 · similarity 0.84)
• Divide et Calibra: Multiclass Local Calibration via Vector Quantization (NeurIPS 2026 · similarity 0.84)