Research map / Learning theory and algorithms / Kernels and regression theory
Information-theoretic Generalization Analysis for Expected Calibration Error
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes the estimation bias of binned expected calibration error (ECE) for uniform-width and uniform-mass binning, derives the optimal bin count, and gives information-theoretic (eCMI) generalization bounds for ECE and TCE.
Problem
The ECE is a binned estimator of the true calibration error (TCE), but its estimation bias and generalization behavior were theoretically poorly understood, especially for uniform-width binning (UWB) and for the total bias combining binning and statistical bias.
Approach
Decomposes the total bias into binning bias and statistical bias, uses a reformulation of ECE as a sum of absolute conditional deviations plus a McDiarmid-style exponential moment bound that works for UWB and UMB, and extends the evaluated CMI (eCMI) framework to derive nonvacuous generalization bounds for ECE and TCE.
Key results
- Optimal bin count B = O(n^{1/3}) yields total bias O(1/n_te^{1/3}) for both UWB and UMB under Lipschitz assumption.
- On MNIST with n=4000, the reuse-of-training-data recalibration reduces TCE to .0058 vs .0085 for standard recalibration (Adam), with bound value .1444 vs .8475.
- On CIFAR-10 (n=20000, SGLD), reuse recalibration achieves TCE .0089 vs .0109 for standard recalibration, with bound .0865 vs 1.455; the bound is below 1 for an appropriate bin count.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reuse training data for recalibration instead of using an independent recalibration dataset | Recalib. with Adam on CNN · MNIST (n=4000) | TCE: $.0085 \pm .0016$ -> $\mathbf{.0058 \pm .0026}$ | Helped |
| Reuse training data for recalibration instead of using an independent recalibration dataset | Recalib. with Adam on CNN · MNIST (n=4000) | Bound value: $.8475$ -> $\mathbf{.1444 \pm .0000}$ | Helped |
| Reuse training data for recalibration instead of using an independent recalibration dataset | Recalib. with SGLD on CNN · MNIST (n=4000) | TCE: $.0101 \pm .0025$ -> $\mathbf{.0072 \pm .0007}$ | Helped |
| Reuse training data for recalibration instead of using an independent recalibration dataset | Recalib. with SGLD on CNN · MNIST (n=4000) | Bound value: $.8475$ -> $\mathbf{.1444 \pm .0000}$ | Helped |
| Reuse training data for recalibration instead of using an independent recalibration dataset | Recalib. with SGD on ResNet · CIFAR-10 (n=20000) | TCE: $\mathbf{.0139 \pm .0010}$ -> $.0197 \pm .0044$ | Hurt |
| Reuse training data for recalibration instead of using an independent recalibration dataset | Recalib. with SGD on ResNet · CIFAR-10 (n=20000) | Bound value: $1.455$ -> $\mathbf{.0865 \pm .0000}$ | Helped |
| Reuse training data for recalibration instead of using an independent recalibration dataset | Recalib. with SGLD on ResNet · CIFAR-10 (n=20000) | TCE: $.0109 \pm .0012$ -> $\mathbf{.0089 \pm .0006}$ | Helped |
| Reuse training data for recalibration instead of using an independent recalibration dataset | Recalib. with SGLD on ResNet · CIFAR-10 (n=20000) | Bound value: $1.455$ -> $\mathbf{.0865 \pm .0000}$ | Helped |
Surprising findings
- Reusing training data for recalibration can give lower TCE than standard held-out recalibration on CIFAR-10 with SGD, despite the conjectured overfitting.
Limitations
- Analysis restricted to binary classification; multiclass extension unclear.
- Cannot handle higher-order (p-norm) TCE.
- Binning bias remains O(1/B) under Holder continuity, so the rate is limited to n^{-1/3}.
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10 synthetic logistic regression data
Metrics: ECE gap TCE gap bound value TCE
Baselines: Recalibration with separate recalibration dataset (UMB)
Builds on: evaluated conditional mutual information (eCMI) McDiarmid inequality SGLD uniform-mass binning analysis of Vaicenavicius et al.
Closest papers by meaning
- PAC-Bayes Analysis for Recalibration in Classification (ICML 2025 · similarity 0.87)
- A Consistent and Differentiable Lp Canonical Calibration Error Estimator (NeurIPS 2022 · similarity 0.87)
- Reassessing How to Compare and Improve the Calibration of Machine Learning Models (ICLR 2025 · similarity 0.87)
- How Flawed Is ECE? An Analysis via Logit Smoothing (ICML 2024 · similarity 0.86)
- A Benchmark Study on Calibration (ICLR 2024 · similarity 0.85)
- Exactly Tight Information-theoretic Generalization Bounds via Binary Jensen-Shannon Divergence (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.