Research map / Learning theory and algorithms / Fairness
Size-adaptive Hypothesis Testing for Fairness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAFT, a size-adaptive hypothesis-testing framework for fairness auditing that combines a CLT-based Wald test for statistical parity on large subgroups with a Dirichlet-multinomial Bayesian credible-interval test for small intersectional subgroups.
Problem
Threshold-based fairness checks on point estimates ignore sampling error and treat small and large demographic (especially intersectional) subgroups identically, leading to unreliable unfairness conclusions.
Approach
Recast statistical parity checking as a hypothesis test: use an asymptotic normality result (delta method on categorical counts) for Wald intervals when subgroup cell counts are at least 30, and a flat-Dirichlet Bayesian Monte-Carlo credible interval otherwise, so the decision threshold adapts to subgroup size.
Key results
- On Adult, the number of detected fairness violations stabilizes at about 1097 across splits and subgroups for n-tilde of 20 to 1000, supporting n-tilde=30 as the switch threshold.
- The paper reports that threshold-based delta-SP flags violations in small COMPAS subgroups where the SAFT interval cannot reject the null, illustrating disagreement between methods.
- Empirical Bayes priors on Adult shrink credible intervals toward Wald width for small subgroups, while with hundreds or thousands of samples all methods give nearly identical intervals.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use size-adaptive hypothesis tests instead of fixed-threshold point estimates | fairness auditing on COMPAS intersectional groups · COMPAS | consistency across train-test splits | Helped |
| use size-adaptive hypothesis tests instead of fixed-threshold point estimates | fairness auditing on Adult intersectional groups · Adult | fairness violation detection | Mixed |
| use Bayesian inference for small groups rather than relying on a Wald test | SAFT · small intersectional groups | uncertainty calibration | Helped |
| use empirical-Bayes priors instead of a uniform prior (ablation) | Bayesian fairness intervals · Adult: Other, Under 18, Female (n_S=2) | confidence/credible interval: [-0.190, 0.488] -> [-0.240, 0.014] (lambda = 0.5); [-0.190, 0.488] -> [-0.236, 0.037] (lambda = 1); [-0.190, 0.488] -> [ | Mixed |
| set minimum support to 1 rather than 30 (ablation) | SAFT · Adult, all subgroups and train-test splits | fairness violations detected: 1097 -> 1159 | Mixed |
| set minimum support to 2 rather than 30 (ablation) | SAFT · Adult, all subgroups and train-test splits | fairness violations detected: 1097 -> 1144 | Mixed |
| set minimum support to 5 rather than 30 (ablation) | SAFT · Adult, all subgroups and train-test splits | fairness violations detected: 1097 -> 1118 | Mixed |
| set minimum support to 10 rather than 30 (ablation) | SAFT · Adult, all subgroups and train-test splits | fairness violations detected: 1097 -> 1098 | No clear effect |
| set minimum support to 20 rather than 30 (ablation) | SAFT · Adult, all subgroups and train-test splits | fairness violations detected: 1097 -> 1097 | No clear effect |
| set minimum support to 40 rather than 30 (ablation) | SAFT · Adult, all subgroups and train-test splits | fairness violations detected: 1097 -> 1097 | No clear effect |
| set minimum support to 50 rather than 30 (ablation) | SAFT · Adult, all subgroups and train-test splits | fairness violations detected: 1097 -> 1096 | No clear effect |
| set minimum support to 100 rather than 30 (ablation) | SAFT · Adult, all subgroups and train-test splits | fairness violations detected: 1097 -> 1098 | No clear effect |
Surprising findings
- The Wald variance plug-in can underestimate uncertainty for small subgroups with extreme proportions, making the Wald test anti-conservative rather than conservative.
- Bootstrap intervals inherit the same failure modes as Wald intervals for small intersectional subgroups and do not improve accuracy.
Limitations
- Bayesian framework requires a prior specification; results depend on the flat or empirical prior choice.
- Empirical validation limited to statistical parity and equal opportunity; other group fairness definitions are left open.
- The n-tilde=30 switch threshold is a rule of thumb rather than derived from theory.
Experimental setup
Datasets: Adult COMPAS German Credit Student Performance
Benchmarks: Adult COMPAS German Credit Student Performance
Metrics: statistical parity difference confidence interval coverage number of fairness violations detected
Baselines: delta-SP fixed threshold gamma-SP bootstrap confidence intervals
Builds on: Kearns et al. gamma-statistical parity Foulds et al. intersectional fairness Dirichlet-multinomial delta method
Closest papers by meaning
- Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness (NeurIPS 2025 · similarity 0.88)
- Intersectional Fairness via Mixed-Integer Optimization (NeurIPS 2026 · similarity 0.87)
- Fairness-Accuracy Trade-Offs: A Causal Perspective (AAAI 2025 · similarity 0.87)
- On the Burden of Achieving Fairness in Conformal Prediction (NeurIPS 2026 · similarity 0.86)
- Identifying Reasons for Bias: An Argumentation-Based Approach (AAAI 2024 · similarity 0.86)
- Fairness under Covariate Shift: Improving Fairness-Accuracy Tradeoff with Few Unlabeled Test Samples (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.