Research map / Label-efficient and robust learning / Recognition and long-tail
"Who experiences large model decay and why?" A Hierarchical Framework for Diagnosing Heterogeneous Performance Drift
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SHIFT, a two-stage hierarchical hypothesis-testing framework that first detects subgroups with large performance decay from aggregate covariate/outcome shifts and then explains them with sparse variable-subset-specific shifts.
Problem
Performance drift of ML models under distribution shift is non-uniform across subgroups; existing methods either explain average performance change or find subgroups without explaining how the decay arises by variable-specific shifts, and often rely on strong assumptions or large data.
Approach
Decompose subgroup performance change into aggregate covariate and outcome shifts, test via omnibus hypothesis tests (McMD-style maximum conditional-mean discrepancy) with sample splitting and one-step debiasing, then test whether sparse variable-subset-specific shifts explain the decay, without assuming the causal graph or parametric models.
Key results
- In synthetic outcome-shift setup 1a, SHIFT achieves power 0.56 versus 0.26 for KCI and 0.06 for MMD; in covariate setup 1b, 0.94 versus 0.0 (KCI) and 0.46 (MMD).
- In the insurance study, the SHIFT-targeted model update reaches AUC 74.8 overall versus 73.0 for the non-targeted update and 69.2 for the original model, and performs best on all listed subgroups.
- In readmission transfer, the top feature (num ED encounters) flagged by SHIFT for both covariate and outcome shifts reflected a differing data extraction procedure; after correcting it the aggregate covariate test is no longer significant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SHIFT instead of KCI or MMD for aggregate outcome-shift detection | Logistic regression · Setup 1a (outcome shift) | power: 0.26 (0.16, 0.4) -> 0.56 (0.42, 0.7); 0.06 (0.02, 0.16) -> 0.56 (0.42, 0.7) | Helped |
| Use SHIFT instead of KCI or MMD for aggregate covariate-shift detection | Logistic regression · Setup 1b (covariate shift) | power: 0.0 (0.0, 0.0) -> 0.94 (0.84, 0.98); 0.46 (0.32, 0.6) -> 0.94 (0.84, 0.98) | Helped |
| Target the detected subgroup and demographic variables rather than retraining everyone on all variables | Insurance coverage MLP · Insurance coverage (overall population) | AUC: 73.0 -> 74.8 | Helped |
| Target the detected subgroup and demographic variables rather than retraining everyone on TE-VIM features | Insurance coverage MLP · Insurance coverage (overall population) | AUC: 73.0 -> 74.8 | Helped |
| Target the detected subgroup and demographic variables rather than retraining everyone on all variables | Insurance coverage MLP · Insurance coverage (Org subgroup) | AUC: 67.3 -> 67.7 | Helped |
| Target the detected subgroup and demographic variables rather than retraining everyone on TE-VIM features | Insurance coverage MLP · Insurance coverage (Org subgroup) | AUC: 64.9 -> 67.7 | Helped |
| Apply the targeted SHIFT update instead of the non-targeted update | Insurance coverage MLP · Insurance coverage (Non-T subgroup) | AUC: 41.8 -> 66.4 | Helped |
| Apply the targeted SHIFT update instead of the TE-VIM update | Insurance coverage MLP · Insurance coverage (TE-VIM subgroup) | AUC: 63.7 -> 65.0 | Helped |
| Correct the extraction of the num ED encounters feature | Readmission GBT · Readmission prediction across hospitals | subgroup accuracy decay due to covariate shift | Helped |
Surprising findings
- Standard covariate-shift tests can flag shifts in features unrelated to the loss due to collider bias from conditioning on subgroup membership, requiring a modified covariate test that filters uncorrelated features.
- Non-targeted model updates inadvertently decayed performance in other subgroups in the insurance study.
Limitations
- Experiments primarily on tabular data; text application is only a preliminary exploration on CivilComments embeddings.
- Tests are conservative due to sample splitting, and aggregate tests are restricted to a singleton detector in theory.
- Requires domain experts to set minimum subgroup prevalence and tolerance; candidate sparse shifts may not explain decay, requiring alternative explanations.
Experimental setup
Datasets: Census health insurance data (Nebraska, Louisiana) hospital readmission data (academic and safety-net hospitals) CivilComments (WILDS)
Benchmarks: synthetic setups 1a/1b/2/3 insurance coverage prediction readmission prediction CivilComments
Metrics: power type-I error p-values accuracy decay AUC
Baselines: KCI MMD TE-VIM ParamY ParamLoss KS test Score test
Builds on: Maximum Mean Discrepancy double/debiased machine learning causal mediation analysis TE-VIM
Closest papers by meaning
- Estimating Model Performance Under Covariate Shift Without Labels (NeurIPS 2025 · similarity 0.84)
- A hierarchical decomposition for explaining ML performance discrepancies (NeurIPS 2024 · similarity 0.84)
- Identifying Predictions That Influence the Future: Detecting Performative Concept Drift in Data Streams (AAAI 2025 · similarity 0.84)
- A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective (NeurIPS 2025 · similarity 0.84)
- What Fits (Into Few Tokens) Doesn't Overfit: Compression and Generalization in ML Research Agents (NeurIPS 2026 · similarity 0.84)
- Evaluating Model Bias Requires Characterizing its Mistakes (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.