Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Provable Uncertainty Decomposition via Higher-Order Calibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces higher-order calibration and k-snapshot based post-hoc and direct methods so that a predictor's aleatoric/epistemic uncertainty decomposition carries provable real-world semantics, demonstrated on CIFAR-10H.
Problem
Predictive uncertainty decompositions into aleatoric and epistemic components from Bayesian and ensemble models lack formal semantics tying them to the real-world data distribution; standard calibration only constrains total uncertainty.
Approach
Define higher-order calibration: a predictor outputting mixtures over label distributions is calibrated if its predicted mixture equals the Bayes mixture over each level set. Achieve a tractable relaxation (k-th order calibration) by learning from k-snapshots (k independent labels per instance), via direct multiclass training over snapshot space or post-hoc empirical mixture estimation per partiti
Key results
- Calibrating with k-snapshots yields increasingly accurate average aleatoric uncertainty estimates on CIFAR-10H, with a significant benefit from going beyond k=2.
- Mean true aleatoric uncertainty on CIFAR-10H is 0.17 (STD 1.2) under Shannon entropy, reflecting nontrivial annotator disagreement.
- Theory: k-th order calibration implies higher-order calibration up to error |Y|/(2 sqrt(k)), and Brier entropy needs only k=2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the k-snapshot size used for post-hoc calibration (ablation) | ResNet calibrated with k-snapshots · CIFAR-10H | aleatoric uncertainty estimation error | Helped |
| use snapshot sizes beyond k=2 (ablation) | ResNet calibrated with k-snapshots · CIFAR-10H | aleatoric uncertainty estimation error | Helped |
Limitations
- Requires multiple independent labels per instance (k-snapshots), not always available.
- Sample complexity scales with |Y|^k, prohibitive for large label spaces or large k.
- Experiments use a coarse partition defined via the predictor output because the true level-set partition is infeasible in high dimensions.
Experimental setup
Datasets: CIFAR-10 CIFAR-10H
Benchmarks: CIFAR-10H
Metrics: aleatoric estimation error Wasserstein calibration error Shannon entropy estimate
Baselines: mutual information decomposition
Builds on: distribution calibration second-order calibration mutual information decomposition conformal prediction
Closest papers by meaning
- The Minimax Rate of Second-Order Calibration (NeurIPS 2026 · similarity 0.88)
- Uncertainty Quantification with the Empirical Neural Tangent Kernel (NeurIPS 2025 · similarity 0.86)
- Beyond One-Hot Labels: Semantic Mixing for Model Calibration (ICML 2025 · similarity 0.86)
- Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks (NeurIPS 2024 · similarity 0.86)
- Second-Order Uncertainty Quantification: A Distance-Based Approach (ICML 2024 · similarity 0.86)
- Parametric ρ-Norm Scaling Calibration (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.