Research map / Learning theory and algorithms / Kernels and regression theory
PAC-Bayes Analysis for Recalibration in Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives the first PAC-Bayes generalization bounds for binning-based expected calibration error (ECE) in multiclass classification and proposes PAC-Bayes recalibration (PBR), a GP-based recalibration objective combining Brier score and KL regularization.
Problem
Theoretical understanding of the bias between top-label calibration error (TCE) and its binning-based ECE estimator, and of generalization of ECE after recalibration, which has been limited to binary classification and lacks guarantees for parametric recalibration methods.
Approach
Applies PAC-Bayes analysis to the binning-based ECE in the multiclass setting, decomposing bias into binning bias and estimation bias with a new concentration inequality for non-i.i.d. ECE terms, yielding an optimizable bound; this motivates a recalibration objective with KL regularization and a differentiable Brier score surrogate for ECE.
Key results
- On CIFAR-100 with WRN-28-10-drop, PBR (Eq. with Brier score) reduces ECE from 0.0568 uncalibrated and 0.0567 GP to 0.0349 at n_re=1000, versus 0.0374 for temperature scaling.
- On MNIST with random forest, PBR reduces ECE from 0.1428 (uncalibrated) and 0.0313 (GP) to 0.0057, with temperature scaling at 0.0062.
- Experiments show a positive correlation between the KL term in the bound and the ECE generalization gap, though the ECE gap is noisy due to the O(n^{-1/3}) convergence rate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PBR instead of GP recalibration | GP recalibration · CIFAR-100, WRN-28-10-drop | ECE: .0567 \pm .0010 -> .0349 \pm .0093 | Helped |
| Use PBR instead of GP recalibration | GP recalibration · CIFAR-100, ResNeXt-8x64d | ECE: .0400 \pm .0010 -> .0310 \pm .0082 | Helped |
| Use PBR instead of GP recalibration | GP recalibration · CIFAR-100, ResNeXt-16x64d | ECE: .0403 \pm .0013 -> .0305 \pm .0056 | Helped |
| Use PBR instead of GP recalibration | GP recalibration · MNIST, Random forest | ECE: .0313 \pm .0056 -> .0057 \pm .0028 | Helped |
| Use PBR instead of GP recalibration | GP recalibration · CIFAR-100, AlexNet | ECE: .2548 \pm .0008 -> .2548 \pm .0007 | No clear effect |
| Use temperature scaling instead of PBR | PBR, AlexNet · CIFAR-100 | ECE: .2548 \pm .0007 -> .0216 \pm .0037 | Helped |
| Use temperature scaling instead of PBR | PBR, 1-layer NN · MNIST | ECE: .0137 \pm .0014 -> .0062 \pm .0026 | Helped |
| Add the Brier score to the Brier + classification objective (ablation) | PBR, MNIST XGBoost · MNIST | ECE: .0037 \pm .0004 -> .0035 \pm .0004 | Helped |
| Add the classification loss to the Brier objective (ablation) | PBR, MNIST Random forest · MNIST | ECE: .0065 \pm .0017 -> .0057 \pm .0028 | Helped |
| Add the classification loss to the Brier objective (ablation) | PBR, CIFAR-100 DenseNet-BC-L190-k40 · CIFAR-100 | ECE: .0630 \pm .0045 -> .0618 \pm .0054 | Helped |
| Add the classification loss to the Brier objective (ablation) | PBR, MNIST 1-layer NN · MNIST | ECE: .0137 \pm .0014 -> .0112 \pm .0019 | Helped |
| Use the Brier + classification objective rather than the Brier objective (ablation) | PBR, CIFAR-100 ResNeXt-16x64d · CIFAR-100 | ECE: .0305 \pm .0056 -> .0321 \pm .0071 | Hurt |
Surprising findings
- Temperature scaling sometimes outperforms GP-based recalibration in ECE when using the theoretically optimal bin number, contrary to prior reports that used B=100.
- GP-based recalibration, including PBR, fails to improve over uncalibrated AlexNet except at small recalibration set sizes.
Limitations
- Binning-based ECE converges slowly at O(n^{-1/3}), making ECE evaluation noisy with limited test data.
- Analysis of CE_K for all classes suffers from the curse of dimensionality (rate O(n^{-1/(K+2)})).
- Bounds rely on Lipschitz continuity assumption and fixed bin count; uniform-mass binning not analyzed.
Experimental setup
Datasets: MNIST CIFAR-100
Benchmarks: MNIST CIFAR-100
Metrics: ECE accuracy KL vs ECE gap rank correlation (Pearson, Kendall)
Baselines: GP-based recalibration Temperature scaling Uncalibrated
Builds on: PAC-Bayes bounds Gaussian process recalibration (variational inference) temperature scaling information-theoretic generalization analysis
Closest papers by meaning
- Information-theoretic Generalization Analysis for Expected Calibration Error (NeurIPS 2024 · similarity 0.87)
- Controlling Multiple Errors Simultaneously with a PAC-Bayes Bound (NeurIPS 2024 · similarity 0.85)
- Recursive PAC-Bayes: A Frequentist Approach to Sequential Prior Updates with No Information Loss (NeurIPS 2024 · similarity 0.85)
- Reassessing How to Compare and Improve the Calibration of Machine Learning Models (ICLR 2025 · similarity 0.85)
- Learning via Surrogate PAC-Bayes (NeurIPS 2024 · similarity 0.84)
- The Minimax Rate of Second-Order Calibration (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.