Research map / Learning theory and algorithms / Prediction and decision losses
Conformity Score Averaging for Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes conformal score averaging, which learns optimal linear weights over multiple conformity scores (THR, APS, RANK) with four data-splitting schemes (VFCP, EFCP, DLCP, DLCP+), and gives VC-based finite-sample coverage and efficiency bounds.
Problem
Single conformity score functions limit the efficiency and informativeness of conformal prediction sets for multi-class classification.
Approach
Linearly combine multiple conformity scores with weights chosen by minimizing average prediction set size on a held-out split, using four data-splitting strategies (two new: DLCP and DLCP+) and VC-theory bounds to justify coverage and efficiency.
Key results
- On CIFAR-100 at alpha=0.01, the VFCP-split weighted method reaches 0.990 coverage with mean set size 13.78, versus 26.40 for RAPS and 40.22 for APS (single-score baselines).
- At alpha=0.05 on CIFAR-100, EFCP/DLCP/DLCP+ reach about 3.75 set size versus 3.89 for VFCP and 5.04 for SAPS and 19.55 for APS.
- The weighted method achieves guaranteed coverage (about 99% at alpha=0.01) while producing the smallest prediction sets among compared score functions across CIFAR-10 and CIFAR-100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine score functions using optimized weights instead of using APS alone | Conformal prediction · CIFAR-100, α = 0.01 | size: 40.217 (1.786) -> 13.782 (1.114) | Helped |
| Combine score functions using optimized weights instead of using THR alone | Conformal prediction · CIFAR-100, α = 0.05 | size: 4.519 (0.381) -> 3.890 (0.266) | Helped |
| Use the EFCP split rather than VFCP (ablation) | Weighted score averaging · CIFAR-100, α = 0.01 | size: 13.782 (1.114) -> 13.306 (0.464) | Helped |
| Use the DLCP split rather than VFCP (ablation) | Weighted score averaging · CIFAR-100, α = 0.01 | size: 13.782 (1.114) -> 13.298 (0.461) | Helped |
| Use the DLCP+ split rather than VFCP (ablation) | Weighted score averaging · CIFAR-100, α = 0.01 | size: 13.782 (1.114) -> 13.299 (0.459) | Helped |
| Use EFCP, DLCP, or DLCP+ rather than VFCP (ablation) | Weighted score averaging · CIFAR-100, α = 0.05 | size: 3.890 (0.266) -> 3.754 (0.096); 3.890 (0.266) -> 3.752 (0.097); 3.890 (0.266) -> 3.753 (0.097) | Helped |
| Weighted combination of score functions | Conformal prediction on CIFAR-10 · CIFAR-10 | size | Helped |
| Weighted combination of score functions | Conformal prediction on CIFAR-100 · CIFAR-100, α ≤ 0.02 | size | Helped |
| Weight scores from multiple models instead of using an individual model | Conformal prediction on CIFAR-10 and CIFAR-100 · CIFAR-10 and CIFAR-100 | size | Helped |
Surprising findings
- THR dominates on CIFAR-10, where the learned weighting places nearly all weight on THR, so score averaging gives limited gains there.
Limitations
- Theory for DLCP and DLCP+ holds only under conditions where the test set influences weight selection, so coverage is not strictly guaranteed for those variants.
- Current theory and method are restricted to classification; extension to regression requires new tools.
- Weight selection uses grid search, which may not scale with many score functions.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 MNIST Fashion-MNIST ImageNet-Val
Benchmarks: CIFAR-10 CIFAR-100 MNIST Fashion-MNIST ImageNet-Val
Metrics: coverage average prediction set size
Baselines: THR APS RANK RAPS SAPS VFCP EFCP Synergy Conformal Prediction individual models (ResNet-56, ShuffleNetV2, VGG16-BN, RepVGG-A2, MobileNetV2)
Builds on: split conformal prediction Yang & Kuchibhotla 2024 (VFCP/EFCP) APS RAPS SAPS
Closest papers by meaning
- Conformal Prediction for Ensembles: Improving Efficiency via Score-Based Aggregation (NeurIPS 2025 · similarity 0.89)
- CBMA: Improving Conformal Prediction through Bayesian Model Averaging (ICLR 2025 · similarity 0.88)
- Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration (NeurIPS 2024 · similarity 0.88)
- Optimal transport-based conformal prediction (ICML 2025 · similarity 0.88)
- Conformal Classification with Equalized Coverage for Adaptively Selected Groups (NeurIPS 2024 · similarity 0.87)
- Hierarchical Conformal Classification (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.