Research map / Probabilistic and scientific modeling / Molecules and proteins
Rectifying Conformity Scores for Better Conditional Coverage
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Rectified Conformal Prediction (RCP), which transforms any conformity score with a learned conditional quantile estimate so that split conformal keeps exact marginal coverage while improving approximate conditional coverage.
Problem
Split conformal prediction guarantees only marginal coverage; exact conditional coverage is infeasible without distributional assumptions, and existing score-transformation methods need full conditional distribution estimates that are costly and hard to estimate, especially for multi-output response
Approach
Rectify any basic conformity score by dividing or shifting it by an estimated conditional quantile of a transformed score (estimated via quantile regression on a held-out split), so the rectified score's quantile is covariate-independent and standard split conformal then yields exact marginal coverage; a bound links conditional coverage to quantile estimation error.
Key results
- On the adversarially selected 10% of the toy data, local coverage drops from 90% (true quantile) to 59% (fully contaminated quantile estimate), while marginal validity holds in all settings.
- RCP variants substantially improve worst-slab coverage over ResCP, PCP and DCP on most of 12 multi-output datasets, with RCP-DCP and RCP-PCP reaching roughly 0.84-0.90 worst-slab coverage where baselines are around 0.72-0.84.
- RCP-DCP obtains smaller median prediction set volume than DCP on most datasets, but larger mean volume, and CPCG is 200-100000 times slower than RCP overall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the true conditional score quantile instead of the fully contaminated estimate (ablation) | RCP · Toy example, adversarial 10% of data | coverage: 59 ± 07 -> 90 ± 01 | Helped |
| Increase contamination of the estimated conditional quantile (ablation) | RCP · Toy example, adversarial 10% of data | coverage: 90 ± 01 -> 84 ± 01 -> 75 ± 03 -> 59 ± 07 | Hurt |
| Use 10-fold cross-validation rather than half the calibration set to estimate the conditional quantile (ablation) | RCP · Multi-output datasets | worst-slab coverage | Helped |
| Use neural-network quantile regression rather than local quantile regression (ablation) | RCP · Multi-output datasets | conditional coverage | Helped |
| Use linear adjustment instead of difference adjustment with DCP (ablation) | RCP-DCP · Multi-output datasets | conditional coverage error | Hurt |
| Use linear rather than difference adjustment with PCP or ResCP (ablation) | RCP-PCP and RCP-ResCP · Multi-output datasets | conditional coverage error | No clear effect |
| Apply RCP score rectification to classic conformal prediction | Multivariate Mahalanobis-score prediction · Multi-output datasets | worst-slab coverage | Helped |
| Apply RCP to PCP, DCP, or ResCP | Multi-output conformal prediction · Real-world datasets | worst-slab coverage | Helped |
| Apply RCP rather than CPCG | DCP-based conformal prediction · Multi-output datasets | worst-slab coverage | Mixed |
| Apply RCP rather than CQR | DCP-based prediction sets · Multi-output datasets | median prediction set volume | Helped |
Surprising findings
- Even coarse estimates of the conditional quantile can substantially improve conditional coverage, but coverage degrades as quantile estimate contamination grows.
- RCP increases mean prediction set volume on several datasets (e.g., ResCP-based variants show very large mean volumes) even while improving median volume and conditional coverage.
Limitations
- Exact conditional coverage is impossible without distributional assumptions, so guarantees are only approximate and depend on quantile estimation error.
- Linear and multiplicative adjustments require restricting the score domain (e.g., positive scores), which can break the theory for some scores such as DCP's negative log-density scores.
- Requires splitting the calibration set (or extra K-fold models) to estimate the quantile, reducing data for the calibration step.
Experimental setup
Datasets: households scm20d rf1 scm1d meps_21 meps_19 meps_20 house bio blog_data calcofi taxi
Benchmarks: households scm20d rf1 scm1d meps_21 meps_19 meps_20 house bio blog_data calcofi taxi
Metrics: worst-slab coverage conditional coverage error marginal coverage median log prediction set volume mean and median prediction set volume computation time
Baselines: ResCP PCP DCP SLCP CQR CPCG
Builds on: conformalized quantile regression normalized conformity scores distributional conformal prediction probabilistic conformal prediction localized conformal prediction
Closest papers by meaning
- Probabilistic Conformal Prediction with Approximate Conditional Validity (ICLR 2025 · similarity 0.93)
- Co-optimization for Adaptive Conformal Prediction (NeurIPS 2026 · similarity 0.91)
- Backward Conformal Prediction (NeurIPS 2025 · similarity 0.90)
- Robust Conditional Conformal Prediction via Branched Normalizing Flow (NeurIPS 2026 · similarity 0.90)
- A Unified Comparative Study with Generalized Conformity Scores for Multi-Output Conformal Regression (ICML 2025 · similarity 0.90)
- Kandinsky Conformal Prediction: Beyond Class- and Covariate-Conditional Coverage (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.