Research map / Learning theory and algorithms / Fairness
Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that thresholding a prediction score into a binary predictor can amplify or ameliorate group disparities, decomposes the disparity via a new margin complement term into causal pathways, and proposes weak/strong business necessity criteria with an audit algorithm.
Problem
Fairness criteria satisfied by a probabilistic prediction score are not preserved after thresholding it into a binary prediction or decision, so disparities can be amplified without being detected in causal terms.
Approach
Introduces the margin complement (the difference between the thresholded predictor and the score), decomposes the total variation of the optimal 0/1 predictor into direct, indirect, and spurious effects from the score and from the margin complement, shows under the standard fairness model that score effects equal outcome effects, and defines weak and strong business necessity accordingly.
Key results
- In a toy hiring example, a 2% outcome disparity (49% vs 51% hire rate) becomes a 100% disparity after thresholding at 0.5, illustrating bias amplification.
- On COMPAS, the direct effect of race on the predictor is nonzero (x-DE(ŝ^NP) ≠ 0) while the effect on the true outcome Y is not statistically different from zero, which violates the court's stated business-necessity requirements.
- Theory shows the score-level causal effects equal the outcome-level effects under the standard fairness model, so the total variation of Ŷ splits into outcome-inherited terms and margin-complement terms.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Threshold scores of 0.49 and 0.51 at 1/2 | Hiring example · Hiring example | group disparity in positive predictions: 2% -> 100% | Mixed |
| Use thresholded binary predictions instead of prediction scores | Optimal prediction · Binary classification | fairness criteria after thresholding | Hurt |
| Threshold the score to form the optimal 0/1 predictor | Hiring example · Hiring example | group disparity attributable to thresholding: 98% | Hurt |
| Apply the score-to-decision decomposition to the MIMIC-IV triage setting | Optimal 0/1 predictor · MIMIC-IV | direct-effect disparity contribution | Mixed |
| Apply the business-necessity audit to COMPAS predictions | Northpointe prediction score · COMPAS | direct-effect fairness requirement | Hurt |
| Use the Northpointe prediction score rather than the true outcome | COMPAS · COMPAS | indirect-effect contribution | Helped |
| Threshold the prediction score and include its margin-complement contribution | Northpointe prediction score · COMPAS | disparity along the direct effect | Hurt |
| Apply the score-to-decision decomposition to salary-bonus decisions | Prediction score thresholded at 0.5 · Census 2018 | direct-effect disparity contribution | Hurt |
| Use the margin-complement contribution along the indirect effect | Prediction score thresholded at 0.5 · Census 2018 | indirect-effect disparity contribution | No clear effect |
Surprising findings
- Satisfying independence or sufficiency for a score S does not guarantee these criteria hold after simple thresholding at 1/2.
- A small outcome disparity in Y can be amplified into a very large disparity in the thresholded predictor Ŷ.
Limitations
- Relies on the standard fairness model and its lack-of-confounding assumptions, which may not hold in practice; partial identification could relax them but is not developed.
- Results depend on the chosen threshold and on the causal diagram assumed for each dataset.
Experimental setup
Datasets: MIMIC-IV COMPAS Census 2018
Benchmarks: MIMIC-IV COMPAS Census 2018
Metrics: direct, indirect, and spurious effects on Y, S, M and Ŷ sample influence
Builds on: standard fairness model path-specific effects disparate impact doctrine
Closest papers by meaning
- Fairness-Accuracy Trade-Offs: A Causal Perspective (AAAI 2025 · similarity 0.91)
- Reconciling Predictive and Statistical Parity: A Causal Approach (AAAI 2024 · similarity 0.91)
- Causal Fairness under Unobserved Confounding: A Neural Sensitivity Framework (ICLR 2024 · similarity 0.90)
- Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification (AAAI 2024 · similarity 0.89)
- Understanding Fairness and Prediction Error through Subspace Decomposition and Influence Analysis (NeurIPS 2025 · similarity 0.89)
- The Fragility of Fairness: Causal Sensitivity Analysis for Fair Machine Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.