Research map / Label-efficient and robust learning / Recognition and long-tail
Truthful Calibration Errors for Multi-Class Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Constructs truthful squared-binned calibration errors for linear properties (classwise and full multiclass), a corrected truthful confidence error, and shows they preserve Blackwell/decision-theoretic ordering and give more stable model rankings across bin counts than non-truthful binned ECE on CIFAR-100 checkpoints.
Problem
Standard binned calibration errors (ECE) are non-truthful: predictors can lower measured error by distorting probabilities, and their rankings of models flip when the number of bins changes.
Approach
Replace the l1 binned residual with its squared l2 version over quantile bins; the bias-variance decomposition of squared binned residuals makes truthful reporting optimal for any linear property, and a vanishing correction term makes the confidence error truthful.
Key results
- Corrected confidence error ConfCE^corr with quantile bins has within-model Spearman 0.691 (20 bins vs 2k bins) with classification error, versus 0.045 for the non-truthful ConfCE at 20 bins.
- Classwise truthful LinCE^cw has within-model Spearman 0.887 with classification error at 20 bins and 0.884 at 5 bins, versus -0.277 for non-truthful ConfECE at 5 bins.
- Non-truthful ConfECE and ConfCE show within-model rank correlation near zero or negative across bin counts (e.g., -0.009 for ConfECE at 20 bins), indicating ranking flips.
Surprising findings
- Non-truthful binned confidence errors can flip their trend with predictive loss when the bin count changes, with within-model Spearman near zero or negative at some bin counts.
Limitations
- Theory applies only to approximately calibrated predictors; ranking guarantees may fail under substantial miscalibration, e.g., post-trained language models.
- Experiments are limited to CIFAR-100 image classifiers with temperature scaling.
- Full multiclass calibration is covered by theory but only classwise specialization is evaluated empirically.
Experimental setup
Datasets: CIFAR-100
Benchmarks: CIFAR-100 test split
Metrics: ConfECE ConfCE LinCE classwise Spearman rank correlation classification error
Baselines: ConfECE ConfCE (non-truthful binned confidence error)
Builds on: Haghtalab et al. truthful calibration measures Hartline et al. quantile-binned calibration error temperature scaling expected calibration error
Closest papers by meaning
- Truthfulness of Calibration Measures (NeurIPS 2024 · similarity 0.88)
- Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling (AAAI 2025 · similarity 0.87)
- The Minimax Rate of Second-Order Calibration (NeurIPS 2026 · similarity 0.86)
- Parametric ρ-Norm Scaling Calibration (AAAI 2025 · similarity 0.86)
- Testing Calibration in Nearly-Linear Time (NeurIPS 2024 · similarity 0.86)
- Improving Multi-Class Calibration through Normalization-Aware Isotonic Techniques (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.