Research map / Learning theory and algorithms / Prediction and decision losses
Truthfulness of Calibration Measures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that existing calibration measures (ECE, smooth calibration, distance from calibration, etc.) have large truthfulness gaps, and introduces Subsampled Smooth Calibration Error (SSCE), which is sound, complete, and (O(1),0)-truthful, with an O(sqrt(T)) adversarial forecaster.
Problem
Calibration measures for sequential probability forecasts should reward truthful prediction (the forecaster should minimize expected penalty by predicting the true conditional expectation), but existing measures can be gamed, leaving large truthfulness gaps.
Approach
Subsample a random subset of time steps uniformly and evaluate smooth calibration error on the subsampled sequence; the expectation over subsets dampens the benefit of strategic deviations, yielding a measure that is sound, complete, and approximately truthful, with a martingale/chaining analysis.
Key results
- SSCE is (c,0)-truthful for a universal constant c, and is complete and sound for binary outcomes.
- Existing measures such as ECE and MSR have 0-Omega(T) truthfulness gaps; smooth calibration, distance from calibration, interval and Laplace-kernel calibration have 0-Omega(sqrt(T)) gaps; U-calibration has an O(1)-Omega(sqrt(T)) gap.
- A deterministic forecaster achieves O(sqrt(T)) SSCE against adaptive adversaries.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Subsample time steps before evaluating smooth calibration error | Smooth Calibration Error · Sequential prediction | truthfulness: $(O(1), 0)$-truthful | Helped |
| Use truthful predictions instead of a strategic forecasting strategy | Expected Calibration Error · Constructed product distribution | ECE penalty: 0-$\Omega(T) | Hurt |
| Use truthful predictions instead of a strategic forecasting strategy | Maximum Swap Regret · Constructed product distribution | maximum swap regret: 0-$\Omega(T) | Hurt |
| Use truthful predictions instead of a strategic forecasting strategy | Smooth Calibration Error · Constructed product distribution | smooth calibration error: 0-$\Omega(\sqrt{T}) | Hurt |
| Use truthful predictions instead of a strategic forecasting strategy | Distance from Calibration · Constructed product distribution | distance from calibration: 0-$\Omega(\sqrt{T}) | Hurt |
| Use truthful predictions instead of a strategic forecasting strategy | Interval Calibration Error · Constructed product distribution | interval calibration error: 0-$\Omega(\sqrt{T}) | Hurt |
| Use truthful predictions instead of a strategic forecasting strategy | Laplace-Kernel Calibration Error · Constructed product distribution | Laplace-kernel calibration error: 0-$\Omega(\sqrt{T}) | Hurt |
| Use truthful predictions instead of a strategic forecasting strategy | U-Calibration Error · Constructed product distribution | U-calibration error: $O(1)$-$\Omega(\sqrt{T})$ | Hurt |
| Evaluate forecasts with a proper scoring rule | Proper Scoring Rules · Sequential prediction | completeness: $\Omega(T)$ | Hurt |
Surprising findings
- Proper scoring rules induce perfectly truthful measures but are not complete: even on i.i.d. Bernoulli trials the truthful forecaster incurs Omega(T) penalty.
- Almost all previously studied calibration measures have large truthfulness gaps, so they can be gamed by forecasters deliberately deviating from the true conditional distribution.
Limitations
- Truthfulness guarantee is stated for the forecaster that knows the true conditional distribution; the results only establish approximate (constant-factor) truthfulness rather than exact truthfulness.
- The SSCE is defined with a constant-factor loss and an additive term in the adversarial bound compared to the adversarial O(sqrt(T)) rate; the analysis is theoretical with no experiments.
Experimental setup
Metrics: truthfulness gap completeness soundness expected penalty
Baselines: expected calibration error maximum swap regret smooth calibration error distance from calibration interval calibration error Laplace-kernel calibration error U-calibration error proper scoring rules
Builds on: smooth calibration error distance from calibration Freedman's inequality Dudley chaining
Closest papers by meaning
- Truthful Calibration Errors for Multi-Class Prediction (NeurIPS 2026 · similarity 0.88)
- Testing Calibration in Nearly-Linear Time (NeurIPS 2024 · similarity 0.87)
- Testable and Actionable Calibration for Full Swap Regret (NeurIPS 2026 · similarity 0.86)
- Conditional Forecasts and Proper Scoring Rules for Reliable and Accurate Performative Predictions (NeurIPS 2025 · similarity 0.85)
- Calibration without labels in multiple testing (NeurIPS 2026 · similarity 0.85)
- Calibration without Ground Truth (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.