Research map / Learning theory and algorithms / Prediction and decision losses
Generalized Venn and Venn-Abers Calibration with Applications in Conformal Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends Venn and Venn-Abers calibration from binary classification to generic loss functions, adds Venn multicalibration, and applies it to conformal quantile prediction with quantile-conditional coverage.
Problem
Point calibrators (histogram binning, isotonic regression) only guarantee calibration asymptotically and can be unreliable in finite samples; existing Venn/Venn-Abers calibration is limited to binary classification and regression, and lacks finite-sample multicalibration for generic losses.
Approach
Wraps any in-sample calibrated point calibrator in a Venn procedure: for each candidate imputed outcome, recalibrate on the augmented calibration set and collect the resulting predictions at the test context into a set guaranteed (marginally, finite-sample) to contain a perfectly calibrated prediction; applies this to generic losses via isotonic regression (Venn-Abers), offset-loss multicalibratio
Key results
- Venn-Abers conformal quantile intervals achieve ~0.90 marginal coverage on all six datasets and the lowest CCE on five of six (e.g., Bike 0.019 vs CQR 0.031, Concrete 0.035 vs 0.037 for CQR, Community 0.028 vs 0.031).
- Venn-Abers intervals are wider than CQR on several datasets (e.g., Concrete width 17 vs 16, Community 0.40 vs 0.38), reflecting the cost of finite-sample set calibration.
- Oracle Venn-calibrated mean predictions reduce multicalibration error versus point-calibrated predictions on all six datasets, e.g., Community 0.0055 vs 0.0209 and Star 0.0078 vs 0.0113.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Venn-Abers quantile calibration instead of CQR | XGBoost-based conformal intervals · Bike, Bio, Star, Meps, Concrete, Community | conditional calibration error (CCE): 0.031 -> 0.019; 0.020 -> 0.017; 0.026 -> 0.024; 0.020 -> 0.018; 0.037 -> 0.035; 0.031 -> 0.028 | Helped |
| Use Venn-Abers instead of CQR | XGBoost-based conformal intervals · Bike, Bio, Star, Meps, Concrete, Community | marginal coverage: 0.90 vs 0.90 on all six datasets | No clear effect |
| Use Venn-Abers instead of CQR | XGBoost-based conformal intervals · Bike, Bio, Star, Meps, Concrete, Community | average interval width: 83 vs 98; 12 vs 14; 620 vs 780; 2.4 vs 2.7; 9.4 vs 16; 0.22 vs 0.38 (uncalibrated vs CQR); Venn-Abers: 100, 14, 780, 2.8 | Mixed |
| Use Venn multicalibration (oracle) instead of point-calibrated offset regression | XGBoost median regression · Bike, Bio, Star, Meps, Concrete, Community | multicalibration error: 0.0015 vs 0.0015; 0.0100 -> 0.0094; 0.0113 -> 0.0078; 0.0017 -> 0.0016; 0.0081 -> 0.0064; 0.0209 -> 0.0055 | Mixed |
| Use Venn multicalibration (oracle) instead of the uncalibrated model | XGBoost median regression · Bike, Bio, Star, Meps, Concrete, Community | multicalibration error: 0.0019 -> 0.0015; 0.0073 -> 0.0094; 0.0098 -> 0.0078; 0.0032 -> 0.0016; 0.0077 -> 0.0064; 0.0099 -> 0.0055 | Mixed |
Surprising findings
- Uncalibrated intervals miss the 90% target badly on some datasets (e.g., 0.71 coverage on Concrete), while the Venn-based methods are calibrated.
- The Venn-calibrated oracle mean prediction is worse than the uncalibrated model on Bio (0.0094 vs 0.0073), an exception to its general improvement.
Limitations
- Venn set calibration must be run separately for each test context, and for continuous outcomes requires discretizing the outcome space, which is approximate.
- Finite-sample guarantees are marginal; conditional calibration holds only asymptotically, with rates O_p(n^{-2/3}) for Venn-Abers and O_p(k(n)log(n/k(n))/n) for binning.
- Quantile-loss results need the conformity scores to be almost surely distinct from calibrated predictions; the weaker lower-bound case is not pursued.
Experimental setup
Datasets: MEPS Concrete Community STAR Bike Bio
Benchmarks: MEPS Concrete Community STAR Bike Bio
Metrics: marginal coverage conditional calibration error (CCE) average interval width multicalibration error prediction set width
Baselines: Uncalibrated CQR Marginal CP VM (Mondrian Venn) Calibrated (offset linear regression)
Builds on: Venn-Abers calibration Venn calibration conformalized quantile regression multicalibrated conformal prediction Mondrian conformal prediction
Closest papers by meaning
- Self-Calibrating Conformal Prediction (NeurIPS 2024 · similarity 0.94)
- The Minimax Rate of Second-Order Calibration (NeurIPS 2026 · similarity 0.86)
- On Temperature Scaling and Conformal Prediction of Deep Classifiers (ICML 2025 · similarity 0.86)
- Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence (ICLR 2025 · similarity 0.86)
- Reassessing How to Compare and Improve the Calibration of Machine Learning Models (ICLR 2025 · similarity 0.86)
- Co-optimization for Adaptive Conformal Prediction (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.