Research map / Learning theory and algorithms / Fairness
FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FFB, an open-source standardized benchmark and codebase for evaluating in-processing group fairness methods on tabular, image, and text datasets, reporting 45,079 experiments.
Problem
Fair comparison of in-processing group fairness methods is hindered by inconsistent experimental settings (preprocessing, splits, backbones), non-standardized implementations, and limited extensibility of existing fairness packages like AIF360 and FairLearn.
Approach
A PyTorch-style, minimalistic open-source benchmark that unifies data preprocessing, splits, in-processing fairness method implementations, and a broad set of utility and group fairness metrics, plus analyses of dataset bias, trade-off controllability, and training stability.
Key results
- HSIC achieves the best utility-fairness trade-off overall among tested in-processing methods on tabular data, per the authors' analysis.
- Adversarial debiasing (AdvDebias) shows unstable, scattered utility-fairness points and is hard to control via hyperparameters.
- Fairness metrics show larger standard deviation than utility metrics across training, while LAFTR shows the most stable fairness performance; the benchmark totals 45,079 experiments and 14,428 GPU hours.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HSIC instead of the other tested fairness methods | In-processing group fairness methods · Tabular datasets | utility-fairness trade-off | Helped |
| Apply HSIC to image data rather than tabular data | HSIC · Image datasets | utility-fairness trade-off | Hurt |
| Increase fairness-control hyperparameters | Most in-processing fairness methods · Benchmark datasets | demographic parity and ABCC | Helped |
| Increase fairness-control hyperparameters | AdvDebias · Benchmark datasets | utility-fairness trade-off | No clear effect |
| Stop training when the learning rate falls below 1e^{-5} | In-processing fairness methods · Benchmark datasets | fairness metric stability | Helped |
| Increase neural-network parameter count | Models evaluated on Jigsaw · Jigsaw | utility and fairness performance | No clear effect |
| Vary fairness-control hyperparameters | Fairness methods on Jigsaw · Jigsaw | demographic parity and ABCC | No clear effect |
Surprising findings
- Several widely used fairness datasets (e.g., KDDCensus race, ACS-E, ACS-P, ACS-M) show no stable bias under ERM and are not recommended for fairness evaluation.
- Architecture size (ResNet variants from 11.6M to 126.9M parameters) does not significantly influence fairness performance.
Limitations
- Relies on traditional accuracy-based utility metrics, which can be problematic when ground truth is biased; uses multiple utility metrics to partially mitigate this.
- Scope limited to in-processing group fairness for binary classification and binary sensitive attributes.
- Training stopping strategy is based on the authors' experience rather than a principled criterion.
Experimental setup
Datasets: Adult German KDDCensus COMPAS Bank ACS-I ACS-E ACS-P ACS-M ACS-T CelebA UTKFace
Benchmarks: Adult German Bank KDDCensus ACS-I ACS-E ACS-P ACS-M ACS-T CelebA UTKFace Jigsaw
Metrics: acc auc ap f1 dp abcc prule eodd eopp
Baselines: ERM DiffDP DiffEopp DiffEodd PRemover HSIC AdvDebias LAFTR
Builds on: AIF360 FairLearn gradient reversal
Closest papers by meaning
- FRAPPÉ: A Group Fairness Framework for Post-Processing Everything (ICML 2024 · similarity 0.90)
- FedFACT: A Provable Framework for Controllable Group-Fairness Calibration in Federated Learning (NeurIPS 2025 · similarity 0.89)
- Enhancing Group Fairness in Online Settings Using Oblique Decision Forests (ICLR 2024 · similarity 0.89)
- Fairness without Demographics through Shared Latent Space-Based Debiasing (AAAI 2024 · similarity 0.87)
- Fairness without Harm: An Influence-Guided Active Sampling Approach (NeurIPS 2024 · similarity 0.87)
- The Fragility of Fairness: Causal Sensitivity Analysis for Fair Machine Learning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.