Research map / Learning theory and algorithms / Kernels and regression theory
Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SmoothECE, a hyperparameter-free calibration measure computed by Nadaraya-Watson kernel smoothing of residuals with a reflected Gaussian kernel, shown to be a consistent calibration measure, with a matching smoothed reliability diagram and a Python package (relplot).
Problem
Binned ECE and binned reliability diagrams are discontinuous, bin-width sensitive, and not tied to consistent calibration measures, so calibration measures and reliability diagrams do not match.
Approach
Smooth the residuals (y - f) with a reflected Gaussian kernel and define SmoothECE as the fixed point scale sigma* where the smoothed L1 residual equals sigma; the reliability diagram plots the kernel-smoothed outcome at the same scale, so the diagram visually encodes the measure.
Key results
- SmoothECE is proven to satisfy polynomial upper and lower bounds with respect to the calibration distance: (1/2) LDCE <= smECE_* <= 2 sqrt(LDCE).
- smECE_sigma is monotone non-increasing in sigma and is 0 for perfectly calibrated distributions.
- Smoothed reliability diagrams are shown alongside binned diagrams on ImageNet ResNet, solar flare, rain, and synthetic data; no quantitative baseline comparison is reported.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a reflected Gaussian kernel instead of a standard Gaussian kernel (ablation) | Smooth reliability diagrams · Predictions on [0,1] | boundary density bias: approximately factor of 2 | Helped |
| Smooth residuals (y-f) instead of smoothing outcomes y when computing the calibration measure (ablation) | SmoothECE · Calibration distributions | monotonicity with bandwidth | Helped |
| Increase the reflected-Gaussian smoothing bandwidth | SmoothECE · Calibration distributions | SmoothECE | Helped |
Surprising findings
- Smoothing the outcomes (the natural reliability diagram estimator) gives a measure that can grow with added smoothing, e.g. a perfectly calibrated distribution yields about 0.79 sigma, so the residual smoothing is needed for the measure.
Limitations
- Generic method may be outperformed by specialized tools when predictions take few discrete values (e.g. rain forecasts in multiples of 10%).
- Bootstrapped bands reflect regressor sensitivity to resampling, not confidence intervals for the true calibration function.
- Theoretical bounds are polynomial, so the consistency constants are loose (sqrt relation).
Experimental setup
Datasets: ImageNet validation set Solar flare forecasts (DAFFS, 731 days) Finland daily precipitation forecasts (Tampere 2003, 346 days) Synthetic data (1000 uniform samples)
Benchmarks: ImageNet validation solar flare forecasts Finland precipitation forecasts
Metrics: SmoothECE reliability diagram visual comparison bootstrapped confidence bands
Baselines: Binned ECE Laplace kernel calibration error / MMCE binned reliability diagram
Builds on: Consistent calibration measures (Błasiok et al.) Nadaraya-Watson kernel regression Weak calibration error duality
Closest papers by meaning
- How Flawed Is ECE? An Analysis via Logit Smoothing (ICML 2024 · similarity 0.89)
- Testing Calibration in Nearly-Linear Time (NeurIPS 2024 · similarity 0.86)
- Truthfulness of Calibration Measures (NeurIPS 2024 · similarity 0.85)
- Truthful Calibration Errors for Multi-Class Prediction (NeurIPS 2026 · similarity 0.84)
- Reassessing How to Compare and Improve the Calibration of Machine Learning Models (ICLR 2025 · similarity 0.84)
- Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.