Research map / Learning theory and algorithms / Prediction and decision losses
Leveraging Uncertainty Estimates To Improve Classifier Performance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that score estimation bias depends on uncertainty and proposes 2D (score x uncertainty) decision boundary algorithms (EW-DPMT, VW-DPMT, GMT, MIST) to maximize recall at a precision bound, gaining 25-40% recall at high precision over score-only thresholding.
Problem
Score-based thresholds for binary classification are misaligned with true positivity rate when training uses differential class sampling or there is train-test distribution drift, so a single score threshold may be suboptimal for maximizing recall at a precision bound.
Approach
Theoretically analyzes how Posterior Network score bias depends on uncertainty (entropy of Beta posterior) and differential sampling, then chooses a score threshold per uncertainty bin via a 2D binned boundary, solved exactly via DP on equi-weight bins, or approximately via per-uncertainty-level isotonic regression with a single global threshold.
Key results
- On Criteo (tau=3), EW-DPMT and MIST improve recall@90% precision by about 22.7% relative to score-only thresholding (2.2% to 2.7%).
- On Avazu with MC-Dropout uncertainty, EW-DPMT gives +48% relative recall at 70% precision over ST (1.28% to 1.9%).
- MIST matches EW-DPMT decision quality while running far faster (e.g., ~99s vs 11335s at 11 uncertainty bins with 1000 score bins on Criteo).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use score and uncertainty for multi-threshold decision boundaries instead of a score-only threshold | Binary classifier with Posterior Network uncertainty · Criteo, 60% precision | Recall: +0.4% | No clear effect |
| Use score and uncertainty for multi-threshold decision boundaries instead of a score-only threshold | Binary classifier with Posterior Network uncertainty · Criteo, 70% precision | Recall: +1.0% | No clear effect |
| Use score and uncertainty for multi-threshold decision boundaries instead of a score-only threshold | Binary classifier with Posterior Network uncertainty · Criteo, 80% precision | Recall: +3.5% | Helped |
| Use score and uncertainty for multi-threshold decision boundaries instead of a score-only threshold | Binary classifier with Posterior Network uncertainty · Criteo, 90% precision | Recall: +18.2% | Helped |
| Calibrate scores independently by uncertainty level and use a global threshold | Binary classifier with Posterior Network uncertainty · Criteo, 90% precision | Recall: +22.7% | Helped |
| Use equi-weight bins and dynamic programming to select thresholds by uncertainty level | Binary classifier with Posterior Network uncertainty · Criteo, 90% precision | Recall: +22.7% | Helped |
| Use MC-Dropout uncertainty to calibrate scores separately by uncertainty level | Binary classifier with MC-Dropout uncertainty · Criteo, 90% precision | Recall: +2.4% | No clear effect |
| Use MC-Dropout uncertainty to calibrate scores separately by uncertainty level | Binary classifier with MC-Dropout uncertainty · Avazu, 70% precision | Recall: +40% | Helped |
| Use MC-Dropout uncertainty to calibrate scores separately by uncertainty level | Binary classifier with MC-Dropout uncertainty · E-Com, 70% precision | Recall: +1.3% | No clear effect |
| Use MC-Dropout uncertainty, equi-weight bins, and dynamic programming to select thresholds by uncertainty level | Binary classifier with MC-Dropout uncertainty · Avazu, 70% precision | Recall: +48% | Helped |
| Use MC-Dropout uncertainty, equi-weight bins, and dynamic programming to select thresholds by uncertainty level | Binary classifier with MC-Dropout uncertainty · E-Com, 70% precision | Recall: +3.9% | No clear effect |
| Fit separate isotonic regression for each uncertainty level instead of one score-only isotonic regression | MIST versus IST · Criteo and Avazu | Expected Calibration Error | Helped |
Surprising findings
- Precision-recall gains from uncertainty increase with the precision bound, with the largest relative gains at high precision.
- The 2D decision boundary problem with variable-size bins is NP-hard, though the equi-weight variant admits an exact polynomial DP.
Limitations
- Theoretical bias analysis relies on Posterior Network and a specific Beta-Bernoulli generative model.
- Code and data are promised for future release only; proprietary E-Com dataset is not public.
- Gains are reported as relative improvements over a low baseline recall, which can be large in relative terms while small in absolute terms.
Experimental setup
Datasets: Criteo Avazu E-Com (proprietary)
Benchmarks: Criteo Avazu E-Com
Metrics: recall at precision bound paired t-test p-values expected calibration error runtime
Baselines: ST (single score threshold) GMT MIST VW-DPMT MC-Dropout uncertainty variant vanilla isotonic regression (IST)
Builds on: Posterior Network MC-Dropout isotonic regression
Closest papers by meaning
- Practical estimation of the optimal classification error with soft labels and calibration (ICLR 2026 · similarity 0.88)
- Optimal Binary Classification Beyond Accuracy (NeurIPS 2022 · similarity 0.86)
- PU-BENCH: A UNIFIED BENCHMARK FOR RIGOROUS AND REPRODUCIBLE PU LEARNING (ICLR 2026 · similarity 0.86)
- Class-Attribute Priors: Adapting Optimization to Heterogeneity and Fairness Objective (AAAI 2024 · similarity 0.86)
- Don’t Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget (ICML 2024 · similarity 0.85)
- What Does It Take to Build a Performant Selective Classifier? (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.