Research map / Architectures and efficiency / Neural architecture search
A Benchmark Study on Calibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a large calibration dataset over ~117k NATS-Bench architectures (CNNs plus 11 ViTs) on CIFAR-10/100 and ImageNet16-120, evaluating ~100 calibration measures to study generalization, robustness, metric reliability, post-hoc scaling, accuracy, bin size, and architecture effects.
Problem
Neural network calibration has been studied mostly via loss functions and training frameworks, while the calibration properties of architectures, and their interaction with NAS, have received little systematic study due to the high cost of training many models.
Approach
Exploit the NATS-Bench search space and its pre-trained models to evaluate calibration metrics on all 117,702 unique CNN architectures (plus ViTs) at scale, then answer longstanding calibration questions with rank-correlation analyses and a proposed Harmonic Calibration Score (HCS) that balances accuracy and calibration.
Key results
- Calibration rankings of architectures correlate poorly across datasets (e.g., CIFAR-10 vs ImageNet nearly uncorrelated, negative for top-1000 models), so calibration does not transfer across datasets.
- Among top-accuracy models, ECE correlates with several robustness metrics, but AUC on OoD datasets shows no correlation with ECE across models of varying accuracy.
- Post-hoc temperature scaling ECE rankings show near-zero correlation with pre-scaling ECE, and bin size has a larger effect on post-scaling ECE (e.g., doubling at 100 bins, quintupling at 500).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate calibration on a different dataset from the one used to assess the architecture | CNN and vision transformer architectures · CIFAR-10, CIFAR-100, and ImageNet16-120 | Kendall ranking correlation of ECE | Hurt |
| Use OoD AUC as a calibration proxy across models with varying accuracy | NATS-Bench TSS models · CIFAR-10 | correlation with ECE | Hurt |
| Use robustness metrics to assess calibration among high-accuracy models | NATS-Bench TSS models · CIFAR-10, top 1000 accuracy-ranked models | correlation with ECE | Helped |
| Use equal-mass classwise calibration error (cwCEem) instead of other calibration metrics | Calibration metric evaluation · ImageNet | Kendall ranking correlation with other metrics | Hurt |
| Apply temperature scaling after evaluating pre-scaling calibration | NATS-Bench TSS models · CIFAR-10 | ranking correlation between pre-ECE and post-ECE | Mixed |
| Increase the bin size for post-temperature-scaling ECE | NATS-Bench models · Calibration evaluation | post-ECE: doubling at a bin size of 100 and quintupling at a bin size of 500 | Hurt |
| Increase model size | NATS-Bench SSS architectures · CIFAR-10 | ECE | Mixed |
| Increase model size | NATS-Bench SSS architectures · ImageNet16-120 | ECE | Hurt |
| Increase model width | NATS-Bench SSS architectures · ImageNet16-120 | ECE: rising from 3% to a substantial 8% | Hurt |
| Select architectures using HCS rather than ECE alone | NATS-Bench TSS models · CIFAR-10 | accuracy-calibration balance | Helped |
Surprising findings
- Calibration does not generalize across datasets even though CIFAR-10 and CIFAR-100 share the same images.
- Equal-mass classwise ECE is negatively correlated with other widely accepted metrics on ImageNet, suggesting it is unreliable.
Limitations
- Findings restricted to image classification and cell-based CNN search spaces; extrapolation to other domains uncertain.
- Analysis focuses on temperature scaling as the only post-hoc calibration method in main experiments.
- Purely empirical; no theoretical analysis of observed calibration patterns.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet16-120 CIFAR-10-C SVHN YFCC100M LAION-400m
Benchmarks: CIFAR-10 CIFAR-100 ImageNet16-120
Metrics: ECE ECE_em cwCE cwCE_em MCE KSCE KDECE MMCE NLL Brier score
Baselines: Regularized Evolution Local Search Random Search temperature scaling
Builds on: NATS-Bench NAS-Bench-201 temperature scaling NAS-Bench-101
Closest papers by meaning
- Revisiting the Calibration of Modern Neural Networks (NeurIPS 2021 · similarity 0.91)
- On Calibration of Modern Neural Networks (ICML 2017 · similarity 0.91)
- Calibration Bottleneck: Over-compressed Representations are Less Calibratable (ICML 2024 · similarity 0.89)
- Reassessing How to Compare and Improve the Calibration of Machine Learning Models (ICLR 2025 · similarity 0.89)
- Uncertainty Weighted Gradients for Model Calibration (CVPR 2025 · similarity 0.88)
- Testing Calibration in Nearly-Linear Time (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.