Research map / Label-efficient and robust learning / Recognition and long-tail
Overcoming Common Flaws in the Evaluation of Selective Classification Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AUGRC, a multi-threshold metric for selective classification built on generalized risk, and shows on the FD-Shifts benchmark (6 datasets, 13 confidence scoring functions) that it reorders CSF rankings relative to AURC.
Problem
Existing multi-threshold metrics for selective classification (chiefly AURC and its variants) fail to align with the task, violate monotonicity, lack interpretability, and are restricted in CSF and error-function flexibility, so they can mislead method benchmarking.
Approach
Replace selective risk (conditional on acceptance) with generalized risk (joint probability of failure and acceptance, averaged over all predictions), yielding AUGRC, which equals (1-AUROC_f)·acc·(1-acc) + ½(1-acc)², is monotonic in accuracy and AUROC_f, and is interpretable as the average risk of undetected failures.
Key results
- AUGRC changes the top-3 CSFs (out of 13) on 5 of 6 datasets compared with AURC.
- Toy example: AURC improves for CSF-3 despite lower AUROC_f and accuracy than CSF-1, a monotonicity violation that AUGRC avoids.
- AUGRC is bounded in [0, 1/2] and has the same computation cost as AURC (~562 µs vs ~564 µs on 1000 scores).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AUGRC instead of AURC for multi-threshold selective-classification evaluation | Selective-classification system benchmarking · 6 FD-Shifts datasets | top-3 confidence-scoring-function ranking | Mixed |
| Evaluate with AUGRC rather than AURC | Selective-classification system benchmarking · FD-Shifts benchmark | confidence-scoring-function rankings | Mixed |
| Use Monte Carlo Dropout with MSR rather than MSR alone | MSR confidence scoring · FD-Shifts benchmark | AUGRC | Helped |
| Apply temperature scaling to MLS | MLS confidence scoring · FD-Shifts benchmark | AUGRC | Mixed |
| Replace selective risk with generalized risk when aggregating across rejection thresholds | AUGRC versus AURC · Selective-classification evaluation | monotonicity with classifier performance and ranking quality | Helped |
| Use AUGRC rather than AURC to compare DG-Res and MCD-PE | Confidence scoring on CIFAR-10 test · CIFAR-10 test | method preference consistent with classification performance and ranking quality | Helped |
| Use AUGRC rather than AURC for evaluation | Selective-classification metric computation · Random scores and failure labels (1000 samples) | evaluation time: 564μs ± 2μs -> 562μs ± 2μs | No clear effect |
Surprising findings
- AURC favors MCD-PE over DG-Res on CIFAR-10 despite DG-Res having higher classification performance and ranking quality, driven by a few high-confidence failures often caused by label ambiguity or incorrect labels.
- F1-AUC decreases with accuracy above roughly 0.56, so it violates monotonicity.
Limitations
- Study only covers image classification benchmarks and the binary 0/1 failure setting for the main analysis.
- Does not extend the benchmark with recent SC methods and does not study the relation to calibration.
- Ranking-based CSF selection is noted as unreliable when rank variability differs across methods.
Experimental setup
Datasets: SVHN CIFAR-10 CIFAR-100 Tiny ImageNet iWildCam BREEDS-Entity-13 CAMELYON-17-Wilds
Benchmarks: FD-Shifts
Metrics: AUGRC AURC e-AURC AUROC_f selective risk rank stability via Wilcoxon signed-rank test
Baselines: AURC e-AURC NAURC F1-AUC AUROC_f OC-AUC AUROC-AURC ARC
Builds on: AURC e-AURC Monte Carlo Dropout DeepGamblers FD-Shifts
Closest papers by meaning
- What Does It Take to Build a Performant Selective Classifier? (NeurIPS 2025 · similarity 0.88)
- A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators (ICML 2025 · similarity 0.86)
- Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs (NeurIPS 2025 · similarity 0.84)
- Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings (ICML 2025 · similarity 0.84)
- Principled Algorithms for Optimizing Generalized Metrics in Binary Classification (ICML 2025 · similarity 0.83)
- Learning with Selectively Labeled Data from Multiple Decision-makers (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.