Research map / Label-efficient and robust learning / Recognition and long-tail
How Flawed Is ECE? An Analysis via Logit Smoothing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes the discontinuities of expected calibration error (ECE) for general distributions and proposes Logit-Smoothed ECE (LS-ECE), a continuous, consistently estimable variant that closely tracks binned ECE on CIFAR and ImageNet classifiers.
Problem
ECE is discontinuous as a function of the predictor and hard to estimate consistently from samples (binned variants are sensitive to bin width), raising the question of whether these pathologies matter for real calibration measurement.
Approach
Characterize exactly where ECE is discontinuous (discrete and general Polish-space settings), then add independent continuous Gaussian noise to the predictor's logit before computing ECE. This yields a continuous functional in L-infinity that admits a kernel-style consistent estimator, and is shown empirically to track binned ECE.
Key results
- On a 2-point synthetic distribution, binned ECE oscillates between about 0 and 1/2 with bin parity, while LS-ECE stays near zero across bin counts.
- On CIFAR-10/100 pretrained ResNet-32, VGG-16 and MobileNetV2, binned ECE stays nearly constant across bin numbers and LS-ECE tracks it closely.
- On ImageNet-1K across nine pretrained architectures, ECE, LS-ECE and smECE take near-identical values across bin numbers and noise scales.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add continuous logit noise before computing calibration error | ECE · Any data distribution | continuity | Helped |
| Use binned ECE while varying the number of bins | Synthetic 2-point distribution · 2-point distribution | ECE: ≈ 0 and ≈ 1/2 | Mixed |
| Use logit-smoothed ECE instead of binned ECE across bin counts and corresponding noise variances | Pretrained CIFAR classifiers · CIFAR-10 and CIFAR-100 test data | calibration error | No clear effect |
| Use logit-smoothed ECE instead of binned ECE across bin counts and noise variances | Pretrained image-classification models · ImageNet-1K-val | calibration error | No clear effect |
| Replace Gaussian noise with uniform noise (ablation) | LS-ECE · CIFAR-10, CIFAR-100, and ImageNet experiments | calibration error comparison with ECE | No clear effect |
| Vary the number of bins used for binned ECE | Synthetic 2-point distribution · 2-point distribution | LS-ECE | No clear effect |
Surprising findings
- ECE can be discontinuous at predictors with no positive-measure level sets for general continuous distributions, contrary to the discrete intuition.
- Binned ECE on the simple 2-point distribution jumps between approximately 0 and 1/2 depending on the parity of the number of bins.
Limitations
- Experiments are preliminary and limited to pretrained image classifiers on CIFAR and ImageNet.
- LS-ECE is not claimed to be a better calibration measure than alternatives; it is primarily a sanity-checking tool.
- Requires the logit transform rho^{-1} to be defined, so predictions of exactly 0 or 1 need tolerance adjustment.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K validation
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1K
Metrics: expected calibration error LS-ECE smECE
Baselines: binned ECE smECE
Builds on: expected calibration error smECE
Closest papers by meaning
- Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing (ICLR 2024 · similarity 0.89)
- Information-theoretic Generalization Analysis for Expected Calibration Error (NeurIPS 2024 · similarity 0.86)
- Reassessing How to Compare and Improve the Calibration of Machine Learning Models (ICLR 2025 · similarity 0.84)
- Testing Calibration in Nearly-Linear Time (NeurIPS 2024 · similarity 0.84)
- Truthful Calibration Errors for Multi-Class Prediction (NeurIPS 2026 · similarity 0.84)
- Invertible Logits Transformation for Accuracy-Preserving Post-Hoc Uncertainty Calibration (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.