Research map / Learning theory and algorithms / Prediction and decision losses
False Coverage Proportion Control for Conformal Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoJER, a nonparametric conformal prediction method that controls the false coverage proportion (FCP) with high probability using JER calibration on a distribution-derived template, plus a harmonic-mean aggregation over multiple models.
Problem
Split conformal prediction controls only the expected proportion of non-covered points (FCR), not the FCP with high probability when many test points are predicted simultaneously.
Approach
Builds FCP upper bounds from Joint Error Rate control applied to conformal p-values, with a calibration template derived from the Monte Carlo distribution of conformal p-values (rather than a DKW-type bound), and calibrates after aggregating p-values from several models so that validity holds for any aggregation function.
Key results
- Standard split conformal prediction satisfies the FCP event in as few as ~35% of splits for some model/dataset pairs, while CoJER and Gazin et al. 2024 keep it above 1-delta = 0.9.
- CoJER intervals are on average about 15% larger than standard SCP intervals, versus about 80% larger in worst cases for Gazin et al. 2024 (which is larger than CoJER).
- In the aggregation experiment, CoJER intervals are about 75% shorter than CV+ and about 20% shorter than Bonferroni intersection on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoJER instead of standard Split Conformal Prediction | Conformal prediction on OpenML datasets · 17 OpenML datasets, 30 splits | FCP event control: 35% | Helped |
| Use CoJER instead of Gazin et al. (2024) | Conformal prediction on OpenML datasets · 17 OpenML datasets, 30 splits | relative interval length: ∼15% larger than standard SCP; ∼80% larger than SCP intervals | Helped |
| Use CoJER instead of standard Split Conformal Prediction | Conformal prediction on OpenML datasets · 17 OpenML datasets, 30 splits | FCP event control | Helped |
| Use harmonic-mean aggregation with CoJER instead of CV+ | Aggregated conformal prediction · 17 OpenML datasets, 30 splits; five regression models | relative interval length: ∼75% larger on average than those of CoJER | Helped |
| Use harmonic-mean aggregation with CoJER instead of Bonferroni intersection | Aggregated conformal prediction · 17 OpenML datasets, 30 splits; five regression models | relative interval length: ∼20% larger on average than those of CoJER | Helped |
| Use CoJER harmonic-mean aggregation instead of CV+ or Bonferroni | Aggregated conformal prediction · 17 OpenML datasets, 30 splits; five regression models | FCP event control: 100% | Mixed |
| Use harmonic-mean aggregation rather than arithmetic mean, geometric mean, or quantile aggregation (ablation) | CoJER aggregation · Appendix C aggregation comparison | interval length | Helped |
Surprising findings
- Standard split conformal prediction fails to control FCP in a majority of splits for some datasets and models, despite controlling FCR.
Limitations
- Results shown only for regression; classification not experimentally evaluated.
- Asymptotic JER control holds only as B goes to infinity, with an O(1/sqrt(B)) Monte Carlo error.
- Harmonic mean aggregation does not always yield valid p-values, so the method relies on JER calibration rather than p-value validity.
Experimental setup
Datasets: 17 OpenML regression datasets from Grinsztajn et al. 2022
Benchmarks: 17 OpenML regression datasets from Grinsztajn et al. 2022
Metrics: FCP event coverage (proportion of splits with FCP <= alpha) relative interval length
Baselines: Split Conformal Prediction Gazin et al. 2024 CV+ Bonferroni intersection (Lei et al. 2018)
Builds on: split conformal prediction Gazin et al. 2024 FCP control Blanchard et al. 2020 joint error rate Blain et al. 2023 knockoff JER calibration CV+
Closest papers by meaning
- Coverage-Guaranteed Prediction Sets for Out-of-Distribution Data (AAAI 2024 · similarity 0.90)
- Rectifying Conformity Scores for Better Conditional Coverage (ICML 2025 · similarity 0.89)
- Backward Conformal Prediction (NeurIPS 2025 · similarity 0.88)
- Co-optimization for Adaptive Conformal Prediction (NeurIPS 2026 · similarity 0.88)
- Kandinsky Conformal Prediction: Beyond Class- and Covariate-Conditional Coverage (ICML 2025 · similarity 0.87)
- Volume Optimality in Conformal Prediction with Structured Prediction Sets (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.