Research map / Learning theory and algorithms / Prediction and decision losses
What Does It Take to Build a Performant Selective Classifier?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes the selective-classification gap relative to a perfect-ordering oracle, gives a finite-sample decomposition into five error sources, and argues via synthetic and vision experiments that monotone calibration barely closes it while re-ranking methods (deep ensembles, self-adaptive training) help.
Problem
Practical selective classifiers fall well short of the perfect-ordering (oracle) accuracy-coverage bound, and it is unclear which properties of the learning setup cause this gap in finite-sample settings.
Approach
Defines a coverage-indexed selective classification gap and decomposes it into Bayes noise, approximation error, ranking error, statistical noise, and miscellaneous slack (optimization and distribution shift). It argues monotone post-hoc calibration cannot change the acceptance ordering and so cannot close the ranking term, whereas non-monotone re-ranking methods can.
Key results
- On CIFAR-100 with CNN, deep ensembles reduce E-AURC from 0.086 (MSP) to 0.065, while temperature scaling leaves it at 0.085 despite lowering ECE from 0.142 to 0.008.
- With ResNet-18, self-adaptive training and deep ensembles lower E-AURC from 0.033 (MSP) to 0.028 and 0.026 respectively.
- Theorem bounds the empirical gap by Bayes + approximation + ranking error plus a C*sqrt(log(1/delta)/n) statistical term; monotone calibration leaves the acceptance set and hence the gap unchanged.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply temperature scaling instead of MSP | CNN · CIFAR-100 | ECE: 0.142 -> 0.008 | Helped |
| Apply temperature scaling instead of MSP | ResNet-18 · CIFAR-100 | ECE: 0.052 -> 0.048 | Helped |
| Apply temperature scaling instead of MSP | WideResNet-50 · CIFAR-100 | ECE: 0.066 -> 0.050 | Helped |
| Apply temperature scaling instead of MSP | CNN · CIFAR-100 | E-AURC: 0.086 -> 0.085 | Helped |
| Apply temperature scaling instead of MSP | ResNet-18 · CIFAR-100 | E-AURC: 0.033 -> 0.033 | No clear effect |
| Apply temperature scaling instead of MSP | WideResNet-50 · CIFAR-100 | E-AURC: 0.031 -> 0.032 | Hurt |
| Use Self-Adaptive Training instead of MSP | CNN · CIFAR-100 | E-AURC: 0.086 -> 0.081 | Helped |
| Use Self-Adaptive Training instead of MSP | ResNet-18 · CIFAR-100 | E-AURC: 0.033 -> 0.028 | Helped |
| Use Self-Adaptive Training instead of MSP | WideResNet-50 · CIFAR-100 | E-AURC: 0.031 -> 0.028 | Helped |
| Use deep ensembles instead of MSP | CNN · CIFAR-100 | E-AURC: 0.086 -> 0.065 | Helped |
| Use deep ensembles instead of MSP | ResNet-18 · CIFAR-100 | E-AURC: 0.033 -> 0.026 | Helped |
| Use deep ensembles instead of MSP | WideResNet-50 · CIFAR-100 | E-AURC: 0.031 -> 0.026 | Helped |
Surprising findings
- Temperature scaling drastically improves ECE (0.142 to 0.008 for CNN on CIFAR-100) yet does not reduce the selective classification gap.
- Histogram binning and vector/Dirichlet scaling are reported to consistently degrade AUROC in selective classification.
Limitations
- Error budget terms can interact, so unique attribution of each source is difficult.
- Experiments focus on synthetic and vision benchmarks; the LLM study is only preliminary in the appendix.
- The oracle bound and gap are defined only for 0-1 loss, not asymmetric costs.
Experimental setup
Datasets: two moons CIFAR-10 CIFAR-100 Stanford Cars CIFAR-10N CIFAR-100N CIFAR-10C Camelyon17-WILDS
Benchmarks: CIFAR-100 Stanford Cars CIFAR-10C Camelyon17-WILDS CIFAR-10N CIFAR-100N
Metrics: E-AURC ECE accuracy-coverage curve
Baselines: MSP Temperature scaling Self-Adaptive Training Deep Ensembles SelectiveNet Deep Gamblers
Builds on: Maximum Softmax Probability Self-Adaptive Training Deep Ensembles Temperature scaling E-AURC
Closest papers by meaning
- Overcoming Common Flaws in the Evaluation of Selective Classification Systems (NeurIPS 2024 · similarity 0.88)
- Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings (ICML 2025 · similarity 0.86)
- Provably Reliable Classifier Guidance via Cross-Entropy Control (NeurIPS 2026 · similarity 0.86)
- Leveraging Data Symmetries to Select an Optimal Subset of Training Data under Label Noise (NeurIPS 2026 · similarity 0.85)
- Selective Mixup Helps with Distribution Shifts, But Not (Only) because of Mixup (ICML 2024 · similarity 0.85)
- Plugin estimators for selective classification with out-of-distribution detection (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.