Research map / Label-efficient and robust learning / Recognition and long-tail
Conformal Prediction for Deep Classifier via Label Ranking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAPS, a conformal prediction non-conformity score that keeps only the maximum softmax probability and replaces the other probabilities with a label-rank term, yielding smaller prediction sets and better conditional coverage than APS and RAPS.
Problem
Conformal prediction sets built from miscalibrated softmax probabilities (APS, RAPS) tend to be large, and they adapt poorly to instance difficulty and conditional coverage.
Approach
Discard all softmax probability values except the maximum softmax probability and replace the remaining ones with a constant so the score depends mainly on label ranking; the maximum probability retains instance-wise uncertainty. Theory shows APS without probability values yields set size tied to top-k accuracy.
Key results
- On ImageNet at alpha=0.1, SAPS reduces average set size to 2.98 versus 20.95 for APS and 3.29 for RAPS, with coverage near 0.900.
- On ImageNet, SAPS lowers average ESCV to 0.396 from RAPS's 0.631 (k_r=1 comparison in the appendix/discussion table).
- Removing softmax values for APS (label ranking only) already shrinks sets, e.g., Inception set size drops from 88.18 to 6.33 on ImageNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace APS with SAPS | conformal prediction classifiers · ImageNet, CIFAR-100, and CIFAR-10 at α=0.1 and α=0.05 | set size: 20.95 to 2.98 | Helped |
| replace APS with SAPS | conformal prediction classifiers · ImageNet, CIFAR-100, and CIFAR-10 at α=0.1 and α=0.05 | coverage | No clear effect |
| replace RAPS with SAPS | conformal prediction classifiers · ImageNet, CIFAR-100, and CIFAR-10 at α=0.1 and α=0.05 | set size: 3.29 to 2.98 | Helped |
| replace RAPS with SAPS | conformal prediction classifiers · ImageNet | SSCV: 0.28 to 0.23 | Helped |
| replace RAPS with SAPS | conformal prediction classifiers · ImageNet | ESCV: 0.53 to 0.40 | Helped |
| remove softmax probability values from APS, retaining label ranking (ablation) | APS · ImageNet, Inception | set size: 88.18 to 6.33 | Helped |
| retain only the maximum softmax probability rather than top 5 probabilities (ablation) | SAPS with ResNeXt101 · ImageNet, α=0.1 | set size: 2.42 to 1.80 | Helped |
| retain additional top-k softmax probabilities (ablation) | SAPS · ImageNet, ResNeXt101 | set size: 1.80, 1.94, 2.03, 2.23, 2.42 | Hurt |
| vary ranking weight λ and calibration set size (ablation) | SAPS · ImageNet | set size | No clear effect |
| use confidence calibration by temperature scaling (ablation) | SAPS · ImageNet, ResNet152 | set size | Helped |
| use SAPS instead of RAPS with k_r=1 (ablation) | conformal prediction classifiers · CIFAR-10, α=0.1 | ESCV: 0.138 to 0.089 | Helped |
| use SAPS instead of RAPS with k_r=1 (ablation) | conformal prediction classifiers · CIFAR-10, α=0.1 | set size: 1.62 to 1.63 | Mixed |
Surprising findings
- Using only label ranking (no probability values) in APS yields much smaller prediction sets than vanilla APS across models.
- Retaining only the maximum softmax probability works best; keeping more top-k probability values increases set size.
Limitations
- Requires tuning the ranking-weight hyperparameter lambda on a validation split taken from the calibration set.
- Conditional coverage remains worse on large-scale datasets such as ImageNet than on small ones.
Experimental setup
Datasets: ImageNet CIFAR-10 CIFAR-100 ImageNet-V2
Benchmarks: ImageNet CIFAR-10 CIFAR-100 ImageNet-V2
Metrics: set size marginal coverage each-size coverage violation (ESCV) size-stratified coverage violation (SSCV)
Baselines: APS RAPS
Builds on: Adaptive Prediction Sets (APS) Regularized Adaptive Prediction Sets (RAPS) temperature scaling
Closest papers by meaning
- Conformal Structured Prediction (ICLR 2025 · similarity 0.88)
- Direct Prediction Set Minimization via Bilevel Conformal Classifier Training (ICML 2025 · similarity 0.87)
- Hierarchical Conformal Classification (NeurIPS 2026 · similarity 0.87)
- Conformal Prediction with Learned Features (ICML 2024 · similarity 0.86)
- Conformity Score Averaging for Classification (ICML 2025 · similarity 0.86)
- Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.