Research map / Label-efficient and robust learning / Recognition and long-tail
Estimating Model Performance Under Covariate Shift Without Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PAPE, a density-ratio-weighted extension of confidence-based performance estimation (CBPE) that estimates any confusion-matrix-based metric of a binary classifier under covariate shift without labels, and tests it on about 959 Folktables census cases.
Problem
Estimating post-deployment performance of binary classifiers on unlabeled production data when covariate shift makes confidence-based estimators miscalibrated.
Approach
Uses a domain classifier (density ratio estimation) to weight a post-hoc calibration mapping fitted on labeled reference data, so the calibrated confidence scores are approximately calibrated on the shifted target distribution; the calibrated scores are then plugged into CBPE to estimate any confusion-matrix metric.
Key results
- PAPE achieved NMAE 0.97 and NRMSE 1.28 for accuracy estimation, versus 1.08/1.75 for CBPE and 1.04/1.40 for IW.
- For F1, PAPE reached NMAE 0.90 and NRMSE 1.34, compared with 1.03/2.12 for CBPE and TEST SET at 2.53/8.23.
- PAPE was reported to significantly outperform the second-best method for each metric (paired Wilcoxon signed-rank test, p = 0.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PAPE instead of IW | performance estimation · Accuracy estimation | NMAE: 1.04 -> 0.97 | Helped |
| use PAPE instead of IW | performance estimation · Accuracy estimation | NRMSE: 1.40 -> 1.28 | Helped |
| use PAPE instead of IW | performance estimation · AUROC estimation | NMAE: 1.06 -> 0.99 | Helped |
| use PAPE instead of IW | performance estimation · AUROC estimation | NRMSE: 1.56 -> 1.45 | Helped |
| add likelihood-ratio weighting during calibration (ablation) | CBPE · F1 estimation | NMAE: 1.03 -> 0.90 | Helped |
| use PAPE instead of IW | performance estimation · F1 estimation | NRMSE: 1.74 -> 1.34 | Helped |
| add likelihood-ratio weighting during calibration (ablation) | CBPE · Accuracy estimation | NMAE: 1.08 -> 0.97 | Helped |
| add likelihood-ratio weighting during calibration (ablation) | CBPE · AUROC estimation | NMAE: 1.07 -> 0.99 | Helped |
Surprising findings
- Simple TEST SET baseline is strongly outperformed for F1, indicating large F1 changes in production chunks; ATC and DoC are competitive only for accuracy.
- RT-mod and COT produce estimation errors the authors describe as unusable for the metrics evaluated.
Limitations
- Fails under concept shift (p_s(y|x) != p_t(y|x)).
- Requires support overlap; shifts outside the source support cannot be weighted.
- Needs enough labeled reference data and works best in low-dimensional settings; density-ratio estimation gets harder with dimensionality.
Experimental setup
Datasets: Folktables ACSIncome Folktables ACSPublicCoverage Folktables ACSMobility Folktables ACSEmployment Folktables ACSTravelTime
Benchmarks: Folktables (US census, 2015-2018)
Metrics: NMAE NRMSE accuracy AUROC F1
Baselines: TEST SET CBPE IW ATC DoC COT RT-mod
Builds on: CBPE importance weighting multicalibration ATC Difference of Confidence
Closest papers by meaning
- "Who experiences large model decay and why?" A Hierarchical Framework for Diagnosing Heterogeneous Performance Drift (ICML 2025 · similarity 0.84)
- Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias (NeurIPS 2026 · similarity 0.84)
- Maximum Likelihood Estimation is All You Need for Well-Specified Covariate Shift (ICLR 2024 · similarity 0.83)
- A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data (NeurIPS 2024 · similarity 0.83)
- Fairness under Covariate Shift: Improving Fairness-Accuracy Tradeoff with Few Unlabeled Test Samples (AAAI 2024 · similarity 0.83)
- Leveraging Uncertainty Estimates To Improve Classifier Performance (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.