Research map / Probabilistic and scientific modeling / Causal discovery
Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses causal DAGs of distribution shift and selection bias to show when disaggregated fairness evaluations (performance parity across subgroups) are misleading, and proposes weighted controlled evaluations as conditional independence tests.
Problem
Disaggregated evaluation across subgroups is widely used as evidence of algorithmic unfairness, but performance differences across subgroups can arise from distributional differences and selection bias rather than model misestimation or unfairness.
Approach
Characterize fairness properties and metric stability of Bayes-optimal and arbitrary models under causal DAGs of subgroup heterogeneity (covariate, outcome, label, presentation shift, and selection), and use weighted controlled evaluation (reweighting the population to match a control variable's subgroup distribution) interpreted as a configurable conditional independence test.
Key results
- Bayes-optimal and arbitrary models generally fail to achieve equal performance across subgroups; separation is implied only in the label shift graph with subgroup-agnostic prediction.
- Subgroup-aware prediction improves log-loss overall and for subgroups in all settings except covariate shift (and separable subgroups), while sensitivity and specificity trade off.
- On ACS PUMS, ACSPublicCoverage shows more evidence against Y perp A given X than ACSIncome, and subgroup-agnostic models show sufficiency violation for both tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use subgroup-aware prediction instead of subgroup-agnostic prediction | simulated settings where Y is not conditionally independent of A given X · Simulation study | log-loss | Helped |
| use subgroup-aware prediction instead of subgroup-agnostic prediction | simulated settings where Y is not conditionally independent of A given X · Simulation study | AUC-ROC | Mixed |
| use subgroup-aware prediction instead of subgroup-agnostic prediction | simulated settings where Y is not conditionally independent of A given X · Simulation study | sensitivity and specificity | Mixed |
| use subgroup-aware prediction instead of subgroup-agnostic prediction | simulated settings where Y is not conditionally independent of A given X · Simulation study | net benefit | Helped |
| use subgroup-aware prediction instead of subgroup-agnostic prediction | simulated settings where Y is not conditionally independent of A given X · Simulation study | calibration and sufficiency | Helped |
| use subgroup-aware prediction instead of subgroup-agnostic prediction | simulated settings with minimal overlap in X across subgroups · Simulation study | calibration | No clear effect |
| use subgroup-aware prediction instead of subgroup-agnostic prediction | ACSIncome · ACS PUMS | predictive performance | Helped |
| use subgroup-aware prediction instead of subgroup-agnostic prediction | ACSPublicCoverage · ACS PUMS | predictive performance | Helped |
| control for X in the evaluation | subgroup-agnostic models · ACSIncome | performance differences between subgroups and aggregate population | Helped |
| control for X in the evaluation | subgroup-agnostic models · ACSPublicCoverage | performance differences between subgroups and aggregate population | No clear effect |
| use subgroup-agnostic prediction instead of subgroup-aware prediction | ACSIncome and ACSPublicCoverage · ACS PUMS | sufficiency | Hurt |
| use stratified prediction instead of subgroup-aware prediction with a race/ethnicity covariate | the Other subgroup · ACSIncome | sufficiency violation | Helped |
Surprising findings
- Equal subgroup performance is not a reliable fairness measure even for Bayes-optimal models with representative data reflecting real-world disparities.
- Subgroup-aware prediction can improve performance overall and for subgroups without mitigating sufficiency violations for all subgroups in real data.
Limitations
- Restricted to binary outcomes; findings may not generalize to continuous or multi-class outcomes.
- Results rely on Bayes-optimality assumptions that cannot be assumed or tested in practice.
- Controlled evaluation assumes oracle access to P(A|V); estimation error of the weight model is not analyzed.
Experimental setup
Datasets: synthetic simulated data (causal and anticausal DGPs with selection) ACS PUMS California 5-year person files (ACSIncome, ACSPublicCoverage via folktables)
Benchmarks: ACSIncome ACSPublicCoverage
Metrics: log-loss AUC-ROC sensitivity specificity precision net benefit calibration curves controlled evaluation statistic T_a
Baselines: subgroup-agnostic gradient boosting stratified models shared-space weighting approach
Builds on: causal DAGs for distribution shift impossibility results for fairness criteria (Kleinberg et al. / Pleiss et al.) shared-overlap weighting for pairwise subgroup comparison
Closest papers by meaning
- The Fragility of Fairness: Causal Sensitivity Analysis for Fair Machine Learning (NeurIPS 2024 · similarity 0.89)
- Size-adaptive Hypothesis Testing for Fairness (NeurIPS 2025 · similarity 0.88)
- Fairness-Accuracy Trade-Offs: A Causal Perspective (AAAI 2025 · similarity 0.88)
- Reconciling Predictive and Statistical Parity: A Causal Approach (AAAI 2024 · similarity 0.88)
- FairPFN: A Tabular Foundation Model for Causal Fairness (ICML 2025 · similarity 0.88)
- Causal Fairness under Unobserved Confounding: A Neural Sensitivity Framework (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.