Research map / Learning theory and algorithms / Prediction and decision losses

Reassessing How to Compare and Improve the Calibration of Machine Learning Models

ICLR 2025 · Learning theory and algorithms › Prediction and decision losses · approach: Probabilistic calibration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that trivial recalibration (mean replacement) can look state-of-the-art on calibration metrics alone, and proposes Bregman-divergence-based reporting plus calibration-sharpness diagrams that jointly visualize calibration and generalization error.

Problem

Calibration of classifiers is reported inconsistently: papers report only ECE-type metrics and test accuracy, which can be gamed by trivial recalibration (e.g., constant confidence equal to test accuracy), hiding losses in sharpness/generalization.

Approach

Decompose Bregman divergences (proper scoring rules) into a calibration term and a sharpness gap, prove that confidence calibration error lower-bounds full calibration error for Bregman divergences, and visualize calibration with a sharpness band via kernel regression in calibration-sharpness diagrams.

Key results

• On ImageNet with ViT, mean replacement recalibration (MRR) reaches binned ECE 0.18 (vs 2.59 for temperature scaling) while its NLL rises to 144.66 (vs 56.83 for TS) and MSE to 27.51 (vs 21.95 for TS).
• The same pattern holds across ResNet-50, EfficientNet-B3, ConvNeXt-Tiny, and ResNet-32 on CIFAR-10/100, where MRR yields near-zero ECE but worse NLL and Brier scores.
• Calibration-sharpness diagrams show isotonic regression improves calibration over histogram binning but at the cost of larger sharpness gaps, a trade-off not visible in reliability diagrams.

Measured effects reported in the paper

Surprising findings

• A trivial constant-confidence recalibration (always reporting test accuracy as confidence) attains near-zero ECE and outperforms popular post-hoc methods on calibration metrics alone.
• Temperature scaling and isotonic regression shift predicted confidences toward even higher probabilities.

Limitations

• Calibration-sharpness diagrams are restricted to the confidence calibration setting for visualization, with the full-calibration link established only through a lower bound.
• Experiments use only pretrained image classifiers and a single kernel bandwidth choice in the main text, with 5000-point subsampling for ImageNet visualizations.

Experimental setup

Datasets: ImageNet-1K-Val CIFAR-10 CIFAR-100

Benchmarks: ImageNet-1K-Val CIFAR-10 CIFAR-100

Metrics: ECE ACE SmoothECE NLL MSE/Brier score test accuracy

Baselines: temperature scaling histogram binning isotonic regression SmoothECE visualization

Builds on: reliability diagrams SmoothECE Bregman divergence decomposition of proper scoring rules (grouping loss) temperature scaling histogram binning

Closest papers by meaning

• Variable-Based Calibration for Machine Learning Classifiers (AAAI 2023 · similarity 0.90)
• Testing Calibration in Nearly-Linear Time (NeurIPS 2024 · similarity 0.90)
• A Benchmark Study on Calibration (ICLR 2024 · similarity 0.89)
• Beyond One-Hot Labels: Semantic Mixing for Model Calibration (ICML 2025 · similarity 0.88)
• Divide et Calibra: Multiclass Local Calibration via Vector Quantization (NeurIPS 2026 · similarity 0.87)
• Information-theoretic Generalization Analysis for Expected Calibration Error (NeurIPS 2024 · similarity 0.87)