Research map / Learning theory and algorithms / Prediction and decision losses
A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a population AURC formulation as a reweighted risk with weight -ln(1-G(x)), proposes plug-in estimators with Monte Carlo-derived rank weights, and analyzes their bias, MSE and convergence rate for selective classifiers.
Problem
Empirical AURC is used as a proxy for population AURC in selective classification, but its finite-sample bias, variance and consistency are not well characterized, and AURC is rarely used directly as a training loss.
Approach
Rewrites population AURC as expected loss reweighted by alpha(x) = -ln(1-G(x)), where G is the CDF of the confidence score; estimates alpha via rank-based Monte Carlo (Beta-distributed order statistics), giving O(n ln n) plug-in estimators whose weights are consistent and whose MSE is bounded, and argues these are preferable to the SELE lower bound.
Key results
- Plug-in weight estimators have MSE of order beta/(n(1-beta)+1) and average MSE bounded by O(ln(n)/n), giving a claimed plug-in convergence rate of O(sqrt(ln(n)/n)).
- On CIFAR10 ResNet18 fine-tuning, the alpha-hat estimator reaches population AURC 4.473 (x10^-2) versus 4.967 for cross-entropy training.
- Empirically, the SELE score underestimates population AURC as sample size grows while 2xSELE overestimates it, whereas the plug-in estimators converge to population AURC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with the α-hat estimator rather than CE loss | Pretrained ResNet models · CIFAR10 | population AURC | Helped |
| Fine-tune with the α-hat estimator rather than CE loss | Pretrained VGG models · CIFAR10 | population AURC | Helped |
| Fine-tune with the α-hat estimator rather than CE loss | Pretrained WideResNet28x10 · CIFAR10 | population AURC | Helped |
| Fine-tune with the α-hat estimator rather than CE loss | Pretrained models · CIFAR100 | population AURC | Helped |
| Use SELE fine-tuning rather than CE fine-tuning | Pretrained models · CIFAR10 and CIFAR100 | population AURC | Helped |
| Use the α-hat-prime weight estimator instead of α-hat (ablation) | AURC plug-in estimator · Finite-sample estimation across evaluated datasets | MSE | Mixed |
| Increase the finite-sample batch size | AURC plug-in estimators · ImageNet | MSE | Helped |
| Increase the finite-sample batch size | SELE estimator · Evaluated datasets | MAE | Mixed |
| Use CE loss instead of 0/1 loss | Finite-sample AURC estimators · Evaluated datasets | variance | Hurt |
| Replace the population-rank weight with the SELE weight (ablation) | Finite-sample AURC estimation · Evaluated datasets | weight-estimation bias | Hurt |
| Use 2×SELE as an upper bound for empirical AURC | Empirical AURC · Five-observation counterexample | empirical AURC bound: 2.2833 ≥ 2 | Hurt |
Surprising findings
- The authors show by counterexample that the claimed upper bound 2x SELE >= empirical AURC does not always hold.
- Although the alpha-hat weight estimator has lower theoretical MSE than alpha-hat-prime, its plug-in estimator empirically shows higher MSE.
Limitations
- Analysis assumes i.i.d. sampling and a fixed model; distribution shift and imbalanced data are left to future work.
- The convergence rate proof combines bounds loosely and the O(sqrt(ln n / n)) rate is only asymptotic.
- Experiments use a limited set of pretrained models with MSP as the main CSF.
Experimental setup
Datasets: CIFAR10 CIFAR100 ImageNet Amazon Reviews
Benchmarks: CIFAR10 CIFAR100 ImageNet Amazon Reviews
Metrics: AURC mean absolute error mean squared error bias standard deviation
Baselines: SELE score 2xSELE cross-entropy fine-tuning
Builds on: SELE loss AURC E-AURC
Closest papers by meaning
- Overcoming Common Flaws in the Evaluation of Selective Classification Systems (NeurIPS 2024 · similarity 0.86)
- Positive-unlabeled AUC Maximization under Covariate Shift (ICML 2025 · similarity 0.84)
- Achievable distributional robustness when the robust risk is only partially identified (NeurIPS 2024 · similarity 0.83)
- What Does It Take to Build a Performant Selective Classifier? (NeurIPS 2025 · similarity 0.83)
- Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias (NeurIPS 2026 · similarity 0.82)
- Boosting Test Performance with Importance Sampling--a Subpopulation Perspective (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.