Research map / Learning theory and algorithms / Fairness

Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification

AAAI 2024 · Learning theory and algorithms › Fairness · approach: Post-processing and transport for fairness · 18 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Defines self-consistency (a variance-derived agreement metric over bootstrapped models) as a proxy for arbitrariness in fair binary classification, proposes a bagging-with-abstention ensemble, and shows most fairness benchmarks look close-to-fair once arbitrariness is accounted for.

Problem

Variance across models trained on different subsamples makes many individual predictions arbitrary in fair binary classification, and single-model or few-model cross-validation evaluations can mask this, leading to unreliable fairness conclusions.

Approach

Introduce self-consistency, the probability that two models from the same learning process agree on a test instance, computed via bootstrapping; extend bagging so the ensemble abstains on instances whose self-consistency falls below a threshold kappa, and use it to re-evaluate fairness on standard benchmarks.

Key results

• On Old Adult, the random forest baseline has expected FNR disparity of 6.3%, which simple ensembling reduces to 4.1%, with systematic arbitrariness measured as W1 = 0.127 for Old Adult versus 0.007 for COMPAS.
• On COMPAS, a logistic regression ensemble achieves FPR disparity between 1.8% and 3% while nearly all tasks show close-to-fair metrics after accounting for arbitrariness.
• For HMDA-NY-2017 random forests, simple ensembling yields mean FNR disparity of 1.1% across 10 ensembles.
• Abstention set error on Old Adult is about 40% for simple ensembling, versus 17% for the RF baseline and 8% for simple ensemble prediction sets.

Measured effects reported in the paper

Surprising findings

• Most fair binary classification benchmarks appear close-to-fair once arbitrariness is accounted for, before any fairness intervention is applied.
• Individual models trained on these tasks show radically different group-specific error rates, yet ensembles average this away, so reproducing unfairness in practice is difficult.

Limitations

• Setup deliberately isolates variance from training-data resampling; ignores other non-determinism such as SGD, HPO, and data collection.
• Abstention yields high error on the abstention set and a trade-off between abstention rate and error; kappa is task-dependent.
• Bootstrapping can be biased (pessimistic) relative to expected error.

Experimental setup

Datasets: COMPAS Old Adult South German Credit Taiwan Credit New Adult - CA HMDA 2007-2017

Benchmarks: COMPAS Old Adult South German Credit Taiwan Credit New Adult - CA HMDA-NY-2017 HMDA-TX-2017

Metrics: self-consistency false negative rate disparity false positive rate disparity abstention rate error Wasserstein-1 distance

Baselines: individual models (single-model baseline) simple ensembling super ensembling

Builds on: bagging equality of opportunity model multiplicity

Closest papers by meaning

• Reconciling Predictive and Statistical Parity: A Causal Approach (AAAI 2024 · similarity 0.90)
• Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making (NeurIPS 2024 · similarity 0.89)
• The Fragility of Fairness: Causal Sensitivity Analysis for Fair Machine Learning (NeurIPS 2024 · similarity 0.89)
• Causal Fairness under Unobserved Confounding: A Neural Sensitivity Framework (ICLR 2024 · similarity 0.88)
• Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness (NeurIPS 2025 · similarity 0.87)
• Fairness-Accuracy Trade-Offs: A Causal Perspective (AAAI 2025 · similarity 0.87)