Research map / Label-efficient and robust learning / Recognition and long-tail
ABCFair: an Adaptable Benchmark approach for Comparing Fairness Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ABCFair, an adaptable benchmark pipeline for comparing fairness methods across stage of intervention, sensitive feature format, fairness notion, and output type, evaluated on 10 methods and tabular datasets including a dual-label SchoolPerformance dataset.
Problem
Fairness methods differ in problem setting (stage of intervention, sensitive feature composition, fairness notion, output distribution), so existing benchmarks that rank methods per dataset are hard to compare fairly; the fairness-accuracy trade-off also looks different when evaluated on unbiased la
Approach
A configurable benchmark pipeline (Dataset, FairnessMethod, Evaluator classes) that reports performance under fairness-violation bounds k per configuration of sensitive feature format, fairness notion, and output type, plus evaluation on a dual-label dataset with unbiased test labels.
Key results
- On SchoolPerformance, fairness-accuracy trade-off is reversed on unbiased labels: increasing fairness increases accuracy, while biased-label accuracy decreases.
- Preprocessing methods struggle to reach very low fairness violations; Learning Fair Repr. achieves low violations only at a large performance loss (AUROC about 63%).
- Exponentiated Gradient and Error Parity lose substantial performance when optimizing hard outputs versus soft scores, while Fairret KLproj incurs no loss across violation levels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate accuracy on unbiased rather than biased labels | fairness methods trained on biased SchoolPerformance labels · SchoolPerformance, unbiased test labels | accuracy | Helped |
| Use stronger fairness constraints (lower violation bound k) | fairness methods on ACSPublicCoverage · ACSPublicCoverage | AUROC | Hurt |
| Use more granular sensitive-feature compositions rather than binary groups | fairness evaluation on ACSPublicCoverage · ACSPublicCoverage, intersectional versus parallel groups | AUROC | Hurt |
| Use preprocessing methods to mitigate bias | preprocessing methods on ACSPublicCoverage · ACSPublicCoverage | AUROC: 63.0_{\pm 1.0 } | Mixed |
| Optimize fairness for soft scores rather than hard predictions | Fairret KL_proj, Exponentiated Gradient, and Error Parity · ACSPublicCoverage, soft-output fairness tables | AUROC: 81.2_{\pm 0.1 } | Mixed |
| Measure performance on hard binary predictions for Error Parity | Error Parity · ACSPublicCoverage, hard demographic parity | accuracy: 78.8_{\pm 0.1 } | Helped |
Surprising findings
- Methods performing better on the traditional biased accuracy-fairness trade-off perform worse on unbiased labels.
- Accuracy-fairness synergy appears on unbiased labels: higher fairness coincides with higher accuracy.
Limitations
- Only 10 methods and 6 tabular datasets validated; no other domains.
- Determining fairness strength values k requires many runs; efficient selection left to future work.
- Only between-group fairness considered; in-group fairness and socio-technical aspects not covered.
Experimental setup
Datasets: SchoolPerformance ACSEmployment ACSIncome ACSMobility ACSPublicCoverage ACSTravelTime folktables
Benchmarks: SchoolPerformance ACSPublicCoverage ACSEmployment ACSIncome ACSMobility ACSTravelTime
Metrics: accuracy AUROC fairness violation (max relative deviation of group statisti
Baselines: Data Repairer Label Flipping Prevalence Sampling Learning Fair Representations Fairret Norm LAFTR Prejudice Remover Exponentiated Gradient Error Parity
Builds on: fairret Fair Fairness Benchmark aequitas fairlearn folktables
Closest papers by meaning
- OxonFair: A Flexible Toolkit for Algorithmic Fairness (NeurIPS 2024 · similarity 0.89)
- FairNet: Dynamic Fairness Correction without Performance Loss via Contrastive Conditional LoRA (NeurIPS 2025 · similarity 0.87)
- FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods (ICLR 2024 · similarity 0.87)
- Towards Harmless Rawlsian Fairness Regardless of Demographic Prior (NeurIPS 2024 · similarity 0.86)
- A Sequentially Fair Mechanism for Multiple Sensitive Attributes (AAAI 2024 · similarity 0.86)
- The Fragility of Fairness: Causal Sensitivity Analysis for Fair Machine Learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.