Research map / Label-efficient and robust learning / Recognition and long-tail
Improving Multi-Class Calibration through Normalization-Aware Isotonic Techniques
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes two normalization-aware isotonic regression calibrators for multi-class post-hoc calibration (NA-FIR and SCIR) and evaluates them against parametric and non-parametric baselines on 7 image/text datasets with about 75 models.
Problem
Post-hoc multi-class calibration with isotonic regression underperforms because one-vs-rest decomposition ignores probability normalization, leaving calibrated outputs that do not form valid probability vectors.
Approach
NA-FIR fits a flattened monotone map while optimizing the NLL of the normalized outputs directly; SCIR fits a 2-D isotonic regression over sorted cumulative top-r probability sums and label counts, which yields a fast dynamic-programming solver.
Key results
- NA-FIR achieves the best average NLL rank on most image and text datasets (e.g., rank 1.0 on ImageNet with 100% win rate and 1.2 on CIFAR-10 with 75%).
- SCIR achieves the best average conf-ECE rank on CIFAR-10 (1.0, 100% of models) and CIFAR-100 (1.2, 85%).
- SCIR runs about 120 minutes on ImageNet-1k versus about 1 second on Yelp Review, while NA-FIR takes about 15 minutes on Yelp Review and ImageNet-1k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use normalization-aware flattened isotonic regression instead of FIR | Multi-class post-hoc calibration · 20NG, CIFAR10, CIFAR100, ImageNet, R52 | NLL average rank: 2.8 -> 1.4; 4.0 -> 1.2; 3.3 -> 1.5; 2.0 -> 1.0; 3.2 -> 1.7 | Helped |
| Use normalization-aware flattened isotonic regression instead of linear scaling | Multi-class post-hoc calibration · Most datasets except Food-101 and Yelp Review | NLL | Helped |
| Use NA-FIR instead of FIR | Multi-class post-hoc calibration · NLL and conf-ECE across datasets | conf-ECE | Helped |
| Use SCIR instead of other calibration methods | Multi-class post-hoc calibration · CIFAR10, CIFAR100, Food-101, ImageNet, R52, Yelp Review | conf-ECE | Mixed |
| Use SCIR rather than NA-FIR | Multi-class post-hoc calibration · CIFAR10, CIFAR100 | conf-ECE average rank: 4.8 -> 1.0; 2.5 -> 1.2 | Helped |
| Use TS instead of NA-FIR | Multi-class post-hoc calibration · Food-101 | NLL average rank: 2.5 -> 1.6 | Helped |
| Use MS instead of NA-FIR | Multi-class post-hoc calibration · Yelp Review | NLL average rank: 3.2 -> 1.5 | Helped |
| Use SCIR instead of NA-FIR | Multi-class post-hoc calibration · Food-101 | NLL average rank: 2.5 -> 5.8 | Hurt |
| Use SCIR rather than NA-FIR | Multi-class post-hoc calibration · CIFAR10 | Brier score average rank: 2.2 -> 1.0 | Helped |
| Use normalization-aware isotonic methods rather than IR OvR | Multi-class post-hoc calibration · Across evaluated datasets | generalization | Helped |
Surprising findings
- SCIR consistently yields worse NLL despite strong conf-ECE, which the authors attribute to near-zero probabilities from overlaps in cumulative class probabilities.
- TS remains best on Food-101 for NLL and conf-ECE, and NA-FIR does not beat linear scaling on Food-101 or Yelp Review.
Limitations
- SCIR produces consistently lower NLL, attributed to near-zero probabilities from cumulative overlaps.
- NA-FIR is non-convex and relies on MCMC search without guaranteed global optimality.
- SCIR scales poorly with the number of classes (about 120 minutes on ImageNet-1k).
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Food-101 ImageNet-1k R52 NG20 Yelp Review
Benchmarks: CIFAR-10 CIFAR-100 Food-101 ImageNet-1k R52 NG20 Yelp Review
Metrics: NLL Brier score conf-ECE cw-ECE TECE accuracy wall-clock time
Baselines: Temperature Scaling Vector Scaling Matrix Scaling IR OvR FIR Uncalibrated
Builds on: isotonic regression Flattened isotonic regression / sCW Temperature Scaling M2B calibration framework PAVA
Closest papers by meaning
- Divide et Calibra: Multiclass Local Calibration via Vector Quantization (NeurIPS 2026 · similarity 0.89)
- Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration (NeurIPS 2019 · similarity 0.88)
- Confidence Calibration of Classifiers with Many Classes (NeurIPS 2024 · similarity 0.86)
- Parametric ρ-Norm Scaling Calibration (AAAI 2025 · similarity 0.86)
- Truthful Calibration Errors for Multi-Class Prediction (NeurIPS 2026 · similarity 0.86)
- Invertible Logits Transformation for Accuracy-Preserving Post-Hoc Uncertainty Calibration (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.