Research map / Learning theory and algorithms / Prediction and decision losses
Distribution-Free Calibration Guarantees for Histogram Binning without Sample Splitting
Chirag Gupta, Aaditya Ramdas
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Avoid sample splitting when forming uniform-mass bins and estimating bin biases | Uniform-mass binning on CREDIT · CREDIT | validity V(0.05): 0.63 -> 0.79 | Improved |
| Avoid 50:50 sample splitting | Uniform-mass binning on CREDIT · CREDIT | sample size for (0.1, 0.1) marginal calibration: n = 1000 -> n = 500 | Improved |
| Use uniform-mass binning instead of fixed-width binning or isotonic regression | Calibration on CREDIT · CREDIT | conditional calibration | Improved |
| Average labels within bins rather than scaling-method predictions | UMD versus scaling-binning on CREDIT · CREDIT | validity | Mixed |
| Exclude the boundary score's label from bin-bias estimation (ablation) | UMD · Distribution-free calibration | conditional calibration bound ε: Original adds 1/floor n/B | Improved |
| Include boundary points in bin-bias estimation (ablation) | Modified UMD · Distribution-free calibration | conditional calibration bound ε: + 1/floor n/B | Hurt |
| Add infinitesimal random perturbations to scores and bin biases (ablation) | UMD · Distribution-free calibration | marginal calibration bound ε: sqrt(log(2B/α)/(2(floor n/B - 1))) -> sqrt(log(2/α)/(2(floor n/B - 1))) + δ | Improved |
| Use 10 bins rather than fewer bins (ablation) | UMD · Distribution-free calibration | calibration bound ε | Mixed |
| Use randomized UMD rather than non-randomized UMD when scores may have ties (ablation) | UMD · Distribution-free calibration | validity | Improved |
Closest papers by meaning
- Information-theoretic Generalization Analysis for Expected Calibration Error (NeurIPS 2024 · similarity 0.83)
- Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling (AAAI 2025 · similarity 0.82)
- PAC-Bayes Analysis for Recalibration in Classification (ICML 2025 · similarity 0.82)
- Sampling Binary Data by Denoising through Score Functions (ICML 2025 · similarity 0.82)
- Testing Calibration in Nearly-Linear Time (NeurIPS 2024 · similarity 0.82)
- Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing (ICLR 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
