Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Self-Calibrating Conformal Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Calibrating Conformal Prediction (SC-CP), which extends Venn-Abers calibration to regression and conformalizes it to give calibrated point predictions with prediction-conditional valid intervals.
Problem
Conformal prediction intervals only guarantee marginal coverage, context-conditional coverage is generally infeasible, and point predictions from ML models are often miscalibrated, which degrades interval efficiency and interpretability.
Approach
Extends Venn-Abers calibration from binary classification to regression by iterating isotonic calibration over imputed outcomes, then builds conformal intervals using self-calibrating conformity scores within level sets of the calibrated predictions, yielding calibrated point predictions and prediction-conditionally valid intervals in finite samples.
Key results
- On MEPS, SC-CP achieves the smallest interval widths among compared methods in both settings while attaining the 0.9 target coverage in each racial subgroup.
- Theory shows finite-sample perfect calibration of the Venn-Abers multi-prediction and prediction-conditional coverage at least 1-alpha, with deviation O(polylog(n)/n^{2/3}).
- In synthetic experiments, calibrating the predictor reduces average interval width relative to uncalibrated absolute residual scores as calibration error increases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SC-CP instead of the listed baselines | MEPS, Setting A and Setting B · MEPS, sensitive-attribute subgroups | average interval width | Helped |
| Calibrate point predictions and use self-calibrating conformity scores | SC-CP · MEPS, sensitive-attribute subgroups | coverage | Helped |
| Use SC-CP instead of Kernel or Mondrian* | MEPS, Setting A and Setting B · MEPS, sensitive-attribute subgroups | average interval width | Helped |
| Use SC-CP instead of Marginal, Mondrian, or Kernel | MEPS, Setting A and Setting B · MEPS, sensitive-attribute subgroups | calibration error | Helped |
| Use CQR instead of SC-CP | MEPS, Setting A and Setting B · MEPS, sensitive-attribute subgroups | calibration error | Hurt |
| Use Mondrian CP with 5 or 10 pre-specified bins | Mondrian CP · MEPS, Setting A and Setting B | coverage | Hurt |
| Use CQR rather than SC-CP | MEPS · MEPS, A=1 subgroup | coverage | Hurt |
| Use SC-CP rather than uncond-CP | Synthetic experiments, Setups A and B · Synthetic data, variance quintiles | coverage | Helped |
| Use SC-CP when outcomes have no mean-variance relationship | SC-CP · Synthetic data, Setup C | conditional coverage | No clear effect |
| Replace uncalibrated residual scores with Venn-Abers calibrated scores (ablation) | Vanilla CP · Synthetic Experiment 1 | average interval width | Helped |
| Increase calibration-set size when using Venn-Abers calibrated scores (ablation) | Vanilla CP with Venn-Abers scoring · Synthetic Experiment 1 | average interval width | Helped |
| Use isotonic calibration with too small a calibration set (ablation) | SC-CP · Synthetic Experiment 1 | average interval width | Hurt |
Surprising findings
- Marginal CP and Mondrian CP show poor calibration error gaps and mis-coverage in subgroups, while SC-CP achieves coverage without pre-specified bins.
Limitations
- Intervals satisfy coverage conditional on the unknown oracle calibrated prediction, with the Venn-Abers set used as an approximation.
- Without mean-variance relationship in outcomes, SC-CP adapts poorly to heteroscedasticity and conditional coverage is poor.
- Requires discretizing the outcome space and running per-context procedures, with computational cost scaling with grid size.
Experimental setup
Datasets: MEPS Concrete Community STAR Bike Bio synthetic Beta-Normal data
Benchmarks: MEPS
Metrics: empirical coverage average interval width calibration error within sensitive attribute
Baselines: Marginal CP Mondrian CP Mondrian* CQR Kernel conditional CP
Builds on: Venn-Abers calibration conformal prediction Mondrian conformal prediction full conformal prediction conformalized quantile regression
Closest papers by meaning
- Generalized Venn and Venn-Abers Calibration with Applications in Conformal Prediction (ICML 2025 · similarity 0.94)
- On Temperature Scaling and Conformal Prediction of Deep Classifiers (ICML 2025 · similarity 0.88)
- Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence (ICLR 2025 · similarity 0.88)
- Conformal Prediction with Learned Features (ICML 2024 · similarity 0.88)
- CBMA: Improving Conformal Prediction through Bayesian Model Averaging (ICLR 2025 · similarity 0.88)
- Boosted Conformal Prediction Intervals (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.