Research map / Learning theory and algorithms / Prediction and decision losses
Calibration tests in multi-class classification: A unifying framework
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use asymptotic p-value approximations for kernel calibration-error estimators instead of distribution-free bounds | Kernel calibration tests · Synthetic M1, M2, and M3 | p-value approximation | Helped |
| Use asymptotic tests, particularly the unbiased quadratic estimator | Kernel calibration tests · Uncalibrated synthetic models M2 and M3 | test power | Helped |
| Use distribution-free p-value bounds | Kernel calibration tests · Synthetic M1, M2, and M3 | test power | Hurt |
| Use consistency resampling for the standard ECE estimator | Standard ECE calibration test · Calibrated synthetic model M1 | empirical test error | Hurt |
| Estimate ECE with histogram regression as the number of classes increases | Histogram-based ECE estimator · Synthetic data with increasing number of classes | calibration error estimate | Hurt |
| Use kernel calibration-error estimators instead of histogram-based ECE as the number of classes increases | Kernel calibration-error estimators · Synthetic data with increasing number of classes | calibration error estimate | Helped |
| Use the linear estimator of squared kernel calibration error | Kernel calibration-error estimation · Synthetic M1, M2, and M3 | bias | Helped |
| Use the linear estimator with 100 or 1000 classes | Linear kernel calibration-error estimator · Synthetic data with 100 and 1000 classes | estimator distribution | Mixed |
| Use p-value approximations based on the histogram ECE estimator as the number of classes increases | Histogram-based ECE calibration test · Synthetic data with increasing number of classes | p-value approximation reliability | Hurt |
Closest papers by meaning
- Confidence Calibration of Classifiers with Many Classes (NeurIPS 2024 · similarity 0.88)
- Improving Multi-Class Calibration through Normalization-Aware Isotonic Techniques (ICML 2025 · similarity 0.88)
- Truthful Calibration Errors for Multi-Class Prediction (NeurIPS 2026 · similarity 0.87)
- Divide et Calibra: Multiclass Local Calibration via Vector Quantization (NeurIPS 2026 · similarity 0.86)
- Reassessing How to Compare and Improve the Calibration of Machine Learning Models (ICLR 2025 · similarity 0.85)
- Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.