Research map / Label-efficient and robust learning / Recognition and long-tail

Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs

NeurIPS 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Group-robust classification · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a clipped cross-entropy (bounded DCA log score) evaluation that averages cost-weighted, prior-adjusted net benefit over a logit-uniform range of deployment class prevalences, linking proper scoring rules (Schervish representation) to clinical cost asymmetry and label shift.

Problem

Standard clinical ML metrics (accuracy, AUC-ROC) ignore calibration, uncertainty in deployment class prevalence (label shift), and asymmetric error costs, so they misjudge which scoring functions are reliable for clinical decisions.

Approach

Derive an adjusted log score by using the Schervish representation of proper scoring rules, clipping the score to a bounded class-balance interval, and averaging cost-weighted net benefit uniformly in log-odds of prevalence, giving a closed-form calibration-based metric that accounts for label shift and cost asymmetry.

Key results

• Proposes a closed-form Bounded DCA Log Score averaging prior-adjusted net benefit over a logit-uniform prevalence range, usable with bootstrapped confidence intervals.
• Shows AUC-ROC equals an average of prior-adjusted maximum accuracy over an implicit prevalence distribution induced by the score, for calibrated scores.
• Illustrates on eICU APACHE IV mortality that racial subgroup gaps differ by class balance; the authors note the public subset lacks statistical power to generalize.

Surprising findings

• Accuracy and AUC-ROC can be misleading for clinical deployment because they depend on a fixed class balance or an implicit, model-dependent prevalence distribution.

Limitations

• Cost uncertainty extension yields dilogarithmic expressions that are analytically opaque and computationally intensive.
• Assumes known deployment class prevalence bounds; adaptive prevalence estimation not handled.
• Small public eICU subset lacks statistical power to generalize subgroup conclusions.

Experimental setup

Datasets: eICU (publicly available subset)

Benchmarks: eICU publicly available subset, in-hospital mortality with APACHE IV

Metrics: accuracy AUC-ROC DCA log score (clipped cross entropy) net benefit weighted accuracy

Baselines: accuracy AUC-ROC net benefit weighted accuracy balanced accuracy AUC-PR

Builds on: Schervish representation decision curve analysis prior-adjusted posterior (label shift) AUC-ROC weighted accuracy

Closest papers by meaning

• Overcoming Common Flaws in the Evaluation of Selective Classification Systems (NeurIPS 2024 · similarity 0.84)
• Timely Clinical Diagnosis through Active Test Selection (NeurIPS 2025 · similarity 0.83)
• SAFER: A Calibrated Risk-Aware Multimodal Recommendation Model for Dynamic Treatment Regimes (ICML 2025 · similarity 0.82)
• Hierarchical Divide-and-Conquer for Fine-Grained Alignment in LLM-Based Medical Evaluation (AAAI 2025 · similarity 0.82)
• Adaptive Labeling for Efficient Out-of-distribution Model Evaluation (NeurIPS 2024 · similarity 0.82)
• Positive-unlabeled AUC Maximization under Covariate Shift (ICML 2025 · similarity 0.81)