Research map / Probabilistic and scientific modeling / Causal discovery
A hierarchical decomposition for explaining ML performance discrepancies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HDPD, a nonparametric hierarchical framework that decomposes ML performance gaps across domains into aggregate (baseline, conditional covariate, conditional outcome shift) terms and Shapley-based variable importances with debiased estimators and confidence intervals.
Problem
Explaining why an ML algorithm's performance differs across domains: existing methods either give only aggregate covariate vs. outcome decompositions, or variable-level breakdowns that need known causal graphs or strong parametric assumptions, and rarely provide confidence intervals.
Approach
Unifies aggregate and detailed performance-gap decompositions: aggregate terms are AIPW/debiased estimands, and variable importance is defined as the proportion of variance in cross-strata performance gaps explained by hypothesized s-partial shifts (covariate shifts via a causal ordering among Z, outcome shifts via binned source risk recalibration), with Shapley values computed by subset sampling
Key results
- Debiased ML confidence intervals reach nominal 90% coverage as sample size grows, whereas naive plug-in intervals do not (simulation).
- In the readmission case study, accuracy drops from 70% to 53%; the top conditional covariate contributors are substance-related and mental health diagnoses, while MeanChange flags nearly every variable as shifted.
- Targeted model revisions using top-k features from HDPD achieve the highest performance for k=1 to 3 in simulation comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select top features using HDPD explanations for targeted model revision | Targeted algorithmic revisions · Simulation | performance | Helped |
| Identify top features using HDPD rather than RandomForestAcc | Targeted model revisions · ACS Public Coverage (NE → LA) | AUC | Helped |
| Use HDPD variable importance to explain the conditional covariate shift | Performance gap explanation · Simulation, conditional covariate shift | variable-importance ranking | Helped |
Surprising findings
- Existing variable-importance methods disagree on rankings in the conditional outcome simulation, and the most-shifted variable by logit coefficient (Z2) is not the most important for performance change (Z1 is).
- A standard test of mean shift flags nearly every variable as significantly shifted in the readmission case, offering little diagnostic value.
Limitations
- Assumes overlapping support of covariates across domains.
- Only handles 0-1 loss/accuracy; extensions to AUC and other metrics left to future work.
- Outcome-shift variable importance requires binned source risk (B=20) and a stronger convergence rate condition on binned risk estimates; baseline shift term has no detailed decomposition.
Experimental setup
Datasets: electronic health records from a large safety-net hospital (30-day readmission) American Community Survey (Nebraska source, Louisiana target) simulated data
Benchmarks: simulated shift settings hospital readmission EHR data ACS public coverage data
Metrics: 0-1 accuracy coverage rate of 90% confidence intervals AUC for targeted model revisions
Baselines: MeanChange Oaxaca-Blinder WuShift ParametricChange ParametricAcc RandomForestAcc
Builds on: Oaxaca-Blinder decomposition Shapley values augmented inverse probability weighting debiased machine learning
Closest papers by meaning
- Task-Agnostic Machine-Learning-Assisted Inference (NeurIPS 2024 · similarity 0.84)
- "Who experiences large model decay and why?" A Hierarchical Framework for Diagnosing Heterogeneous Performance Drift (ICML 2025 · similarity 0.84)
- A Survey of Learning Criteria Going beyond the Usual Risk (Abstract Reprint) (AAAI 2024 · similarity 0.84)
- Divide-and-Aggregate Learning for Evaluating Performance on Unlabeled Data (AAAI 2024 · similarity 0.83)
- Data-centric Prediction Explanation via Kernelized Stein Discrepancy (ICLR 2025 · similarity 0.83)
- Local vs. Global Interpretability: A Computational Complexity Perspective (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.