Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Conformal Prediction for Ensembles: Improving Efficiency via Score-Based Aggregation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes conformal score aggregation (CSA), a multivariate-score conformal method that uses data-driven quantile envelopes to build tighter prediction regions for ensembles, applied to classification and predict-then-optimize regression.
Problem
Conformal prediction for ensembles of black-box models: existing conformal aggregation methods merge separately conformalized prediction regions, forgoing shared structure in the conformal scores and yielding overly conservative regions.
Approach
Stack per-predictor non-negative scores into a multivariate score and define a data-driven pre-ordering via nested convex sets built from directional quantile envelopes (estimated on a first calibration split), then calibrate a scalar threshold on a second split, retaining finite-sample coverage while producing smaller regions.
Key results
- On ImageNet at alpha=0.10 (coverage 0.90), CSA yields average set size 34.0 versus 69.1 for the ensemble conformal baseline and 46.1 for VFCP.
- On OpenML regression at alpha=0.05, CSA achieves coverage 0.957 with interval length 7.69 on dataset 361234, versus 17.76 for the ensemble score and 9.20 for C^M.
- On Manhattan traffic routing, CSA coverage is 0.968 versus 0.981 (STEPS) and 0.962 (LDM), with paired t-test p-values of 3.6e-4 and 9.5e-4 for reduced suboptimality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| aggregate ensemble scores with CSA rather than VFCP | ResNet-50, VGG-11, DenseNet-121 ensemble · ImageNet (α=0.10) | average prediction set size: 46.063 (1.089) -> 34.006 (0.924) | Helped |
| aggregate ensemble scores with CSA rather than VFCP | ResNet-50, VGG-11, DenseNet-121 ensemble · ImageNet (α=0.05) | average prediction set size: 78.108 (2.004) -> 59.574 (3.382) | Helped |
| aggregate ensemble scores with CSA rather than VFCP | ResNet-50, VGG-11, DenseNet-121 ensemble · ImageNet (α=0.01) | average prediction set size: 194.691 (4.579) -> 201.32 (46.509) | Hurt |
| use two-stage calibration instead of single-stage calibration (ablation) | CSA regression · OpenML regression tasks | coverage | Helped |
| use two-stage calibration instead of single-stage calibration (ablation) | CSA · ImageNet classification | average prediction set size: 58.424 (1.674) -> 59.574 (3.382) | No clear effect |
| use CSA aggregation rather than conformalizing predictors separately | STEPS and LDM traffic prediction ensemble · Manhattan traffic routing (α=0.05) | expected suboptimality gap proportion | Helped |
Surprising findings
- The single-stage variant (no data splitting for the pre-ordering) fails to maintain coverage on several OpenML tasks, e.g. coverage 0.745 on dataset 361235 at alpha=0.05.
- On some regression tasks, the best individual conformalized predictor (e.g., random forest) outperforms CSA, and CSA is not uniformly best.
Limitations
- Requires splitting the calibration set into two parts, with the first used to define the pre-ordering, which reduces data for the final quantile.
- Predict-then-optimize regression regions must be approximated on a discretized grid, and the approach scales with the product of sample counts across predictors.
- Coverage holds only marginally and the pre-ordering uses sample-based quantile estimation; results are on relatively small test sets.
Experimental setup
Datasets: ImageNet OpenML regression tasks (361234-361249) Manhattan traffic routing dataset (precipitation/traffic)
Benchmarks: ImageNet OpenML regression tasks 361234-361249 Manhattan traffic routing
Metrics: coverage prediction set size prediction interval length expected suboptimality gap proportion
Baselines: VFCP C^M C^R C^U Ensemble conformal score Single-Stage CSA STEPS-based conformal LDM-based conformal
Builds on: conformal prediction conformal model aggregation robust predict-then-optimize (conformal) directional quantiles Danskin's theorem
Closest papers by meaning
- Multi-model Ensemble Conformal Prediction in Dynamic Environments (NeurIPS 2024 · similarity 0.90)
- Conformity Score Averaging for Classification (ICML 2025 · similarity 0.89)
- Optimal transport-based conformal prediction (ICML 2025 · similarity 0.88)
- Kandinsky Conformal Prediction: Beyond Class- and Covariate-Conditional Coverage (ICML 2025 · similarity 0.88)
- A Kernel Nonconformity Score for Multivariate Conformal Prediction (NeurIPS 2026 · similarity 0.87)
- A Unified Comparative Study with Generalized Conformity Scores for Multi-Output Conformal Regression (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.