Research map / Learning theory and algorithms / Prediction and decision losses
Non-exchangeable Conformal Prediction with Optimal Transport: Tackling Distribution Shift with Unlabeled Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives optimal-transport upper bounds on the total coverage gap of split conformal prediction under distribution shift, including a label-free version using unlabeled test data, and uses them to learn calibration weights.
Problem
Split conformal prediction loses marginal coverage when calibration and test data are non-exchangeable due to distribution shift, and existing reweighting fixes need prior knowledge of the shift type or labeled test data.
Approach
Bound the total coverage gap (averaged over all miscoverage rates) by weighted CDF or 1-Wasserstein distances between calibration and test nonconformity-score distributions; obtain label-free bounds by sandwiching the unknown test score distribution with auxiliary distributions (min/max or uniform/model-based label scores) and optimize calibration weights directly on these bounds.
Key results
- On iWildCam, the (min,max) OT variant raised average coverage by more than 7 points and the (f,U) variant by almost 10 points over uncorrected scores, which drop roughly 10% below the 90% target.
- On ImageNet-C severity 5, uncorrected coverage falls to about 30%; the OT weighted variants bring coverage to roughly 71-80% with and without label shift at much smaller set sizes than the RBF conditional-coverage baseline.
- Free-form and weight-function parametrizations perform very similarly across experiments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OT-learned calibration weights instead of uncorrected calibration | Split conformal prediction · ImageNet-C and iWildCam distribution shifts | coverage: from the target coverage of 90%, we drop to an average of ~80% for severity level 1, and to an average of ~30% for sever | Helped |
| Use (f, U) auxiliary score distributions instead of (min, max) (ablation) | OT calibration weighting · iWildCam | coverage: increased coverage by more than 7% on average; improved coverage by almost 10% | Helped |
| Use weight-function parameterization instead of free-form weights (ablation) | OT calibration weighting · iWildCam | coverage | Helped |
| Use free-form weights instead of weight-function parameterization (ablation) | OT calibration weighting · ImageNet-C severity 1 | coverage | Helped |
| Use weighted-CDF objective instead of 1-Wasserstein objective (ablation) | OT calibration weighting · Image classification | coverage | Helped |
| Use 1-Wasserstein objective instead of weighted-CDF objective (ablation) | OT calibration weighting · Regression | coverage | Helped |
| Optimize a restricted coverage-range objective instead of total coverage gap (ablation) | OT calibration weighting · Coverage range [α−, α+] | coverage: only marginal improvements | No clear effect |
| Use learned likelihood-ratio weights instead of uncorrected calibration | Split conformal prediction · iWildCam | coverage | No clear effect |
Surprising findings
- The conditional-coverage RBF baseline hurt coverage on iWildCam (dropping to about 38% under label shift in one setting), possibly due to class imbalance or a mismatched shift class.
- The OT-based methods do not always beat the uncorrected scores; in minimal-shift settings the uncorrected calibration can already give better coverage than the learned weights.
Limitations
- Bounds with auxiliary distributions from min/max or uniform/model label scores can be loose for unknown shifts.
- The (f,U) sandwich relation is not guaranteed and only empirically observed to hold in most cases.
- Unnecessary reweighting can hurt coverage when there is little or no distribution shift.
Experimental setup
Datasets: ImageNet-C iWildCam synthetic regression
Benchmarks: ImageNet-C iWildCam synthetic regression
Metrics: coverage prediction set size
Baselines: likelihood ratio weighting entropy scaled conformal prediction conditional coverage method with RBF kernels uncorrected split conformal
Builds on: split conformal prediction weighted conformal prediction regression-as-classification Dvoretzky-Kiefer-Wolfowitz inequality
Closest papers by meaning
- Non-Exchangeable Conformal Risk Control (ICLR 2024 · similarity 0.90)
- Optimal transport-based conformal prediction (ICML 2025 · similarity 0.89)
- Improving the Statistical Efficiency of Cross-Conformal Prediction (ICML 2025 · similarity 0.88)
- Robust Conditional Conformal Prediction via Branched Normalizing Flow (NeurIPS 2026 · similarity 0.88)
- Conformal Predictions under Markovian Data (ICML 2024 · similarity 0.88)
- On Volume Minimization in Conformal Regression (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.