Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes binomial process modeling with a beta-derived logistic prior family to estimate a continuous calibration curve by maximum likelihood, a consistent calibration metric (TCE_bpm), and a binomial-process simulator for benchmarking calibration metrics.
Problem
Post-hoc confidence calibration estimates are unstable in low-density confidence regions, and existing methods do not principledly integrate prior distributions behind the calibration curve with empirical data; calibration metrics also lack ground truth for evaluation.
Approach
Model the calibration data sampling as a binomial process over all confidence scores, parameterize the calibration curve with a three-parameter logistic prior family derived from beta class-conditional distributions, and fit it by minimizing an exponentiated squared-error likelihood surrogate; then estimate TCE by integrating the fitted curve against a moment-fitted beta confidence density.
Key results
- On simulated data D1, the mean absolute difference from the true calibration curve is 0.0099 for the proposed method versus 0.0233 for the mean of histogram binning.
- On ResNet110 CIFAR-10, the proposed TPM calibration gets the best ECE_bin (0.00458 vs 0.00745 for temperature scaling) and best TCE_bpm (0.00214 vs 0.00649 for isotonic regression).
- The paper reports the proposed calibration error is 50.15% lower on average than the second-place method across six metrics on ResNet110 CIFAR-10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TPM calibration instead of temperature scaling | ResNet110 · CIFAR-10 | ECE_bin: 0.00745 -> 0.00458 | Helped |
| Use TPM calibration instead of isotonic regression | ResNet110 · CIFAR-10 | ECE_debiased: 0.00550 -> 0.00312 | Helped |
| Use TPM calibration instead of spline calibration | ResNet110 · CIFAR-10 | ECE_sweep: 0.00347 -> 0.00146 | Helped |
| Use TPM calibration instead of spline calibration | ResNet110 · CIFAR-10 | KS-error: 0.00430 -> 0.00162 | Helped |
| Use TPM calibration instead of temperature scaling | ResNet110 · CIFAR-10 | smECE: 0.00901 -> 0.00756 | Helped |
| Use TPM calibration instead of isotonic regression | ResNet110 · CIFAR-10 | TCE_bpm: 0.00649 -> 0.00214 | Helped |
| Estimate the calibration curve with binomial process modeling and a prior rather than mean histogram binning | Calibration curve estimation · Simulated calibration data D1 | mean absolute error: 0.0233 -> 0.0099 | Helped |
Surprising findings
- Temperature scaling and Mix-n-Match give identical results on ResNet110 CIFAR-10 across most metrics in the reported table.
- The proposed method is not best on smECE (0.00756 is worse than temperature scaling's 0.00901 only partially; the method is best on smECE in the table but the authors note the calibration metric is not always closest to TCE).
Limitations
- Bayesian averaging over many binning schemes increases computational cost.
- Only closed-set classification is considered; multi-label, open-set, and generative settings are not addressed.
- Selecting the three most representative bins for efficient estimation is left open.
Experimental setup
Datasets: CIFAR-10 logits (ResNet110) CIFAR-100 logits (Wide-ResNet32) ImageNet logits (DenseNet162) Simulated binomial-process datasets from true distributions D1-D5
Benchmarks: CIFAR-10 ResNet110 logits Simulated datasets D1-D5
Metrics: ECE_bin ECE_debiased ECE_sweep KS-error smECE TCE_bpm mean absolute difference (EAD) P-value
Baselines: Temperature scaling Isotonic regression Mix-n-Match Spline calibration Histogram binning ECE_debiased ECE_sweep KS-error smECE LS-ECE
Builds on: histogram binning temperature scaling Platt scaling consistent calibration measures (Błasiok et al.) smooth ECE
Closest papers by meaning
- Testing Calibration in Nearly-Linear Time (NeurIPS 2024 · similarity 0.87)
- Truthful Calibration Errors for Multi-Class Prediction (NeurIPS 2026 · similarity 0.87)
- Parametric ρ-Norm Scaling Calibration (AAAI 2025 · similarity 0.87)
- Reassessing How to Compare and Improve the Calibration of Machine Learning Models (ICLR 2025 · similarity 0.86)
- Confidence Calibration of Classifiers with Many Classes (NeurIPS 2024 · similarity 0.86)
- Beyond One-Hot Labels: Semantic Mixing for Model Calibration (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.