Research map / Learning theory and algorithms / Prediction and decision losses
Non-Exchangeable Conformal Risk Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends conformal risk control to non-exchangeable data by weighting calibration points by relevance, giving a risk bound that adds a total-variation penalty to the target level alpha.
Problem
Conformal risk control guarantees expected monotone losses only under exchangeable data, while real-world data (change points, time series, drift) often violates exchangeability.
Approach
Combines weighted (non-exchangeable) conformal prediction with conformal risk control: calibration points are weighted by data-independent or data-dependent weights, and the expected loss of a new test point is bounded by alpha plus a weighted total-variation term (B-A) sum of w_i d_TV(Z, Z^i), recovering standard CRC when data is exchangeable.
Key results
- On synthetic changepoint and drift settings with alpha=0.2, standard CRC averages risk 0.246 and 0.225 while non-X CRC achieves 0.196 and 0.182, meeting the target.
- On Natural Questions F1 control with alpha=0.3 over 1000 splits, non-X CRC gives smaller average set size (23.0 vs 24.6) at similar test risk (0.30).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use non-exchangeable CRC with weights w_i=0.99^{n+1-i} instead of standard CRC | CRC · Synthetic multilabel, setting 1 (i.i.d.) | average loss: 0.191~/~0.183 -> 0.181~/~0.175 | Helped |
| Use non-exchangeable CRC with weights w_i=0.99^{n+1-i} instead of standard CRC | CRC · Synthetic multilabel, setting 2 (changepoints) | average loss: 0.246~/~0.228 -> 0.196~/~0.183 | Helped |
| Use non-exchangeable CRC with weights w_i=0.99^{n+1-i} instead of standard CRC | CRC · Synthetic multilabel, setting 3 (distribution drift) | average loss: 0.225~/~0.218 -> 0.182~/~0.175 | Helped |
| Use non-exchangeable CRC with weights w_i=0.99^{n+1-i} instead of standard CRC | CRC · Natural Questions | average set size: 24.6 ± 1.83 -> 23.0 ± 1.47 | Helped |
| Use embedding-similarity weights for calibration questions | non-X CRC · Natural Questions | test risk: 0.30 ± 0.015 | No clear effect |
| Use weighted least-squares regression with non-X CRC instead of standard least squares | Electricity monitoring on ELEC2 · ELEC2, original time order | interval width at the desired loss | Helped |
| Use non-exchangeable methods instead of standard CRC + LS during distribution drift | Electricity monitoring on ELEC2 · ELEC2, original time order | loss relative to desired threshold | Helped |
| Use non-exchangeable CRC instead of standard CRC | CRC · ELEC2, randomly permuted data | loss relative to desired threshold | No clear effect |
Limitations
- Theoretical bound is loose when total variation distances are large; weights must be chosen carefully since true TV distances are unknown.
- Data-dependent weights require a modified bound using expectations, and the main guarantee holds for data-independent weights.
- Experiments are limited to synthetic data and two real-world tasks with simple models.
Experimental setup
Datasets: ELEC2 Natural Questions synthetic multilabel time series
Benchmarks: ELEC2 Natural Questions synthetic multilabel time series
Metrics: false negative rate lambda-insensitive absolute loss best token-level F1 risk average set size average risk
Baselines: conformal risk control (CRC) CRC with covariate shift (Gibbs et al. proposition 4)
Builds on: conformal risk control weighted conformal prediction (Barber et al. 2022) split conformal prediction
Closest papers by meaning
- Non-exchangeable Conformal Prediction with Optimal Transport: Tackling Distribution Shift with Unlabeled Data (NeurIPS 2025 · similarity 0.90)
- Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them) (ICML 2024 · similarity 0.89)
- Conformal Risk Control (ICLR 2024 · similarity 0.89)
- Conformal Risk Training: End-to-End Optimization of Conformal Risk Control (NeurIPS 2025 · similarity 0.87)
- Improving the Statistical Efficiency of Cross-Conformal Prediction (ICML 2025 · similarity 0.87)
- Neural Conformal Control for Time Series Forecasting (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.