Research map / Learning theory and algorithms / Prediction and decision losses
Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows conformal prediction validity extends to any joint data distribution via a general weighted-permutation formula, and derives tractable multistep feedback covariate shift CP algorithms evaluated on protein design and active learning.
Problem
Standard conformal prediction guarantees require exchangeability or quasi-exchangeability, which excludes sequential feedback-loop covariate shifts induced by AI/ML agents in active learning and black-box optimization.
Approach
Views conformal prediction as an inverted permutation test and proves coverage for any joint PDF by weighting calibration and test points with permutation-based conditional probabilities; then factorizes the density for multistep feedback covariate shift and truncates to the d most recent steps to get tractable weights.
Key results
- On protein design with 1,000 seeds, MFCS Full CP maintains coverage above target at all five design steps where both One-Step FCS and standard Full CP lose coverage.
- For MLP regressor split CP, MFCS methods with depth d in {3,4} maintain target coverage across all 10 design steps on both fluorescent protein datasets where baselines break down.
- In active learning with GP regressor over 70 steps and 350 seeds, the proposed MFCS Split CP maintains target coverage on all datasets while baselines lose coverage.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MFCS Full CP (d=2) instead of Exchangeable Full CP or One-Step FCS Full CP | Full CP for protein design · Blue and red fluorescent protein design | coverage | Helped |
| Use MFCS Split CP with estimation depth d in {3, 4} instead of the baselines | Split CP with an MLP regressor · Blue and red fluorescent protein design | coverage | Helped |
| Increase MFCS weight-estimation depth d (ablation) | MFCS Split CP with an MLP regressor · Blue and red fluorescent protein design | coverage | Helped |
| Use MFCS Split CP instead of unweighted Split CP, One-Step Split CP, or ACI | Split CP with a GP regressor · Active learning | coverage | Helped |
| Restrict the ML agent's query probability function | Agent-induced MFCS conformal prediction · Agent-induced feedback covariate shift | interval width | Helped |
Surprising findings
- The authors argue the general coverage result was implicit in prior work (Tibshirani et al. 2019) but not stated, suggesting quasi-exchangeability assumptions are not necessary for CP validity.
- Both MFCS Split CP and One-Step FCS occasionally yield infinite-width intervals, which the authors note may be unacceptable in some settings.
Limitations
- Exact general weights are factorial in cost and require knowledge of the density f; practicality relies on factorization and truncation to d steps.
- Results are restricted to symmetric score functions; nonsymmetric score functions are left for future work.
- Prediction intervals can be infinite-width for some settings, and the guarantee is marginal rather than conditional coverage.
Experimental setup
Datasets: blue fluorescent protein design library red fluorescent protein design library combinatorially complete protein library
Benchmarks: in silico protein design (blue and red fluorescent protein) active learning regression tasks
Metrics: empirical coverage median interval width mean squared error on holdout
Baselines: Exchangeable CP One-Step FCS CP Adaptive Conformal Inference Unweighted Split CP
Builds on: weighted conformal prediction (Tibshirani et al. 2019) feedback covariate shift conformal (Fannjiang et al. 2022) full conformal prediction split conformal prediction adaptive conformal inference
Closest papers by meaning
- Conformal Prediction as Bayesian Quadrature (ICML 2025 · similarity 0.90)
- Non-Exchangeable Conformal Risk Control (ICLR 2024 · similarity 0.89)
- An Information Theoretic Perspective on Conformal Prediction (NeurIPS 2024 · similarity 0.89)
- Reliable Uncertainty Quantification in Machine Learning via Conformal Prediction (AAAI 2025 · similarity 0.89)
- Online Conformal Prediction via Online Optimization (ICML 2025 · similarity 0.89)
- Adaptive Conformal Inference by Betting (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.