Research map / Learning theory and algorithms / Prediction and decision losses
Metrics of Calibration for Probabilistic Predictions
Imanol Arrieta-Ibarra, Paman Gujral, Jonathan Tannen, Mark Tygert, Cherie Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use cumulative-difference ECCE statistics instead of binned ECE | calibration metric · synthetic data, growing sample size | ability to separate perfect from imperfect calibration | Improved |
| increase number of observations | ECCE for perfectly calibrated data · synthetic data | ECCE | Improved |
| increase number of observations | ECE for perfectly calibrated data · synthetic data | ECE | No clear effect |
| use ECCE-MAD/sigma_n on perfectly calibrated data | ECCE-MAD · synthetic data | expected value: 1.2533 | No clear effect |
| use ECCE-R/sigma_n on perfectly calibrated data | ECCE-R · synthetic data | expected value: 1.5958 | No clear effect |
| use cumulative plots instead of reliability diagrams | calibration visualization · ImageNet subsets | reveals miscalibration without binning | Improved |
Closest papers by meaning
- Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing (ICLR 2024 · similarity 0.83)
- Truthful Calibration Errors for Multi-Class Prediction (NeurIPS 2026 · similarity 0.81)
- Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling (AAAI 2025 · similarity 0.80)
- How Flawed Is ECE? An Analysis via Logit Smoothing (ICML 2024 · similarity 0.79)
- Testing Calibration in Nearly-Linear Time (NeurIPS 2024 · similarity 0.79)
- Calibration without labels in multiple testing (NeurIPS 2026 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
