Research map / Label-efficient and robust learning / Recognition and long-tail
A Closer Look at AUROC and AUPRC under Class Imbalance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically and empirically refutes the claim that AUPRC is superior to AUROC under class imbalance, showing AUPRC can favor high-prevalence subgroups, and audits 1.5M papers for the origin of the claim.
Problem
Whether AUPRC is a better model-comparison metric than AUROC under class imbalance, and whether AUPRC introduces subgroup disparities when subgroup prevalences differ.
Approach
Derives closed-form expressions for AUROC and AUPRC in terms of FPR weighted uniformly (AUROC) versus inversely by the model's firing rate (AUPRC), showing each mistake correction improves AUROC equally but AUPRC more for high-score mistakes; under calibration and prevalence imbalance, AUPRC provably focuses optimization on the high-prevalence subgroup.
Key results
- On synthetic data, optimizing AUPRC makes the high-prevalence group's AUROC rise while the low-prevalence group's AUROC can decrease under random-permutation optimization, whereas AUROC optimization raises both groups together.
- Across real fairness datasets, the Spearman correlation between overall AUPRC and signed AUROC gap vs. prevalence gap has rho = 0.905 (p = 0.002), indicating AUPRC favors higher-prevalence groups more as the prevalence disparity grows.
- A literature review of over 1.5M arXiv papers found 424 publications making the claim that AUPRC is better than AUROC under class imbalance, 167 of them without citation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| optimize model selection by AUPRC rather than AUROC | Synthetic models with high- and low-prevalence subgroups · Synthetic experiments | Per-group AUROC disparity | Hurt |
| optimize by AUPRC using nearby-score permutations rather than AUROC | Synthetic models with high- and low-prevalence subgroups · Synthetic experiments, nearby-score permutations | Low-prevalence subgroup AUROC | Hurt |
| optimize by AUROC rather than AUPRC | Synthetic models with high- and low-prevalence subgroups · Synthetic experiments | Per-group AUROC | Helped |
| use AUPRC rather than AUROC for model selection | XGBoost models · Real-world fairness datasets with higher prevalence ratios | Correlation with signed AUROC gap | Mixed |
| increase the prevalence disparity between groups | AUPRC-versus-AUROC model selection comparison · Real-world fairness datasets and attributes | Spearman correlation of prevalence gap with AUPRC bias amplification: Spearman's ρ = 0.905, p = 0.002 | Helped |
| correct a higher-score adjacent ranking mistake rather than a lower-score one | AUPRC · Binary classification ranking | AUPRC | Helped |
| correct any adjacent ranking mistake | AUROC · Binary classification ranking | AUROC | Helped |
| select the mistake with the greatest AUPRC improvement | AUPRC on calibrated models with sufficiently imbalanced subgroup prevalence · Two-subgroup classification | Probability the selected mistake is within the higher-prevalence subgroup | Helped |
Surprising findings
- Under random-permutation optimization, optimizing AUPRC can decrease the low-prevalence group's AUROC.
- Of the 257 papers citing a source for the claim, 135 cited papers that do not actually make this claim.
Limitations
- Theorems assume perfect calibration of the model, which is restrictive.
- Real-world prevalence ratios are much lower than the synthetic ratio of 5, so the observed effect is mild.
- Only two subpopulations are analyzed; results extend only to pairwise comparisons.
Experimental setup
Datasets: adult compas lsac mimic
Benchmarks: adult compas lsac mimic
Metrics: AUROC AUPRC per-group AUROC gap per-group AUPRC gap Spearman correlation
Baselines: AUROC optimization AUPRC optimization
Builds on: AUROC AUPRC XGBoost
Closest papers by meaning
- Interplay of ROC and Precision-Recall AUCs: Theoretical Limits and Practical Implications in Binary Classification (ICML 2024 · similarity 0.89)
- Imbalances in Neurosymbolic Learning: Characterization and Mitigating Strategies (NeurIPS 2025 · similarity 0.85)
- Learning under Localized Minority Imbalance (NeurIPS 2026 · similarity 0.85)
- Pairwise AUC Optimization Needs Corrective Power: A Unified View (NeurIPS 2026 · similarity 0.85)
- Balancing the Scales: A Theoretical and Algorithmic Framework for Learning from Imbalanced Data (ICML 2025 · similarity 0.85)
- Positive-unlabeled AUC Maximization under Covariate Shift (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.