Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Conformal Prediction as Bayesian Quadrature
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reframes conformal prediction and conformal risk control as Bayesian decision problems and proposes a Bayesian-quadrature-based bound on posterior expected loss using Dirichlet quantile spacings, which recovers CRC and split conformal as special cases.
Problem
Conformal prediction and conformal risk control give frequentist, marginal guarantees that average over unobserved calibration sets, ignore prior knowledge, and can be violated in a large fraction of individual deployments.
Approach
Places a prior over quantile functions of the loss distribution and uses a worst-case (right-rectangle) quadrature bound with Dirichlet-distributed quantile spacings to get a distribution over the expected loss, whose mean recovers split conformal and CRC; the upper-bound distribution L+ enables data-conditional high-probability guarantees.
Key results
- On synthetic binomial data, CRC exceeded the target risk in 21.20% of 10,000 trials versus 0.03% for the proposed method at beta=0.95.
- On MS-COCO, CRC exceeded the risk threshold in 45.05% of trials, RCPS in 0.0% (set size 3.57), and the proposed method in 5.43% with set size 3.04.
- On synthetic heteroskedastic data, the proposed method had 3.42% failure rate with mean interval length 9.50 versus 46.19% and 7.99 for CRC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Bayesian HPD decision rule instead of CRC | Conformal risk control · Synthetic binomial | relative frequency exceeding target risk: 21.20% -> 0.03% | Helped |
| Use the Bayesian HPD decision rule instead of RCPS | Risk control · Synthetic binomial | relative frequency exceeding target risk: 0.00% -> 0.03% | Hurt |
| Use the Bayesian HPD decision rule instead of split conformal prediction / CRC | Split conformal prediction / CRC · Synthetic heteroskedastic | relative frequency exceeding target risk: 46.19% -> 3.42% | Helped |
| Use the Bayesian HPD decision rule instead of RCPS | Risk control · Synthetic heteroskedastic | mean prediction interval length: 14.29 -> 9.50 | Helped |
| Use the Bayesian HPD decision rule instead of split conformal prediction / CRC | Split conformal prediction / CRC · MS-COCO | relative frequency exceeding target risk: 45.05% -> 5.43% | Helped |
| Use the Bayesian HPD decision rule instead of RCPS | Risk control · MS-COCO | prediction set size: 3.57 -> 3.04 | Helped |
| Use the Bayesian HPD decision rule instead of RCPS | Risk control · MS-COCO | relative frequency exceeding target risk: 0.0% -> 5.43% | Hurt |
Surprising findings
- Conformal risk control's marginal guarantee holds, yet 21.20% of individual calibration-set trials exceed the target risk on synthetic data.
Limitations
- Assumes i.i.d. deployment data matching calibration data.
- Requires a known upper bound B on losses.
- Bounds are conservative since they hold for worst-case priors consistent with the data.
Experimental setup
Datasets: MS-COCO synthetic binomial data synthetic heteroskedastic data
Benchmarks: MS-COCO false negative rate control synthetic binomial synthetic heteroskedastic
Metrics: relative frequency of risk exceeding alpha prediction set size mean prediction interval length
Baselines: Conformal Risk Control RCPS Split Conformal Prediction
Builds on: split conformal prediction conformal risk control Bayesian quadrature risk-controlling prediction sets
Closest papers by meaning
- CBMA: Improving Conformal Prediction through Bayesian Model Averaging (ICLR 2025 · similarity 0.91)
- Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them) (ICML 2024 · similarity 0.90)
- Reliable Uncertainty Quantification in Machine Learning via Conformal Prediction (AAAI 2025 · similarity 0.90)
- An Information Theoretic Perspective on Conformal Prediction (NeurIPS 2024 · similarity 0.90)
- Probabilistic Conformal Prediction with Approximate Conditional Validity (ICLR 2025 · similarity 0.89)
- Non-exchangeable Conformal Prediction with Optimal Transport: Tackling Distribution Shift with Unlabeled Data (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.