Research map / Learning theory and algorithms / Fairness
The Disparate Benefits of Deep Ensembles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that Deep Ensembles raise accuracy/AUROC but can worsen group fairness (SPD, EOD, AOD) because predictive diversity differs across groups, and that Hardt post-processing effectively mitigates this by exploiting the ensembles' better calibration.
Problem
Deep Ensembles are widely used to boost predictive performance, but their impact on group fairness across protected attributes is not well understood.
Approach
Names and analyzes a "disparate benefits" effect: ensembling unevenly improves performance across protected groups, driven by per-group differences in average predictive diversity of members; Hardt group-dependent thresholds then mitigate the fairness loss because ensembles are better calibrated and more threshold-sensitive.
Key results
- Deep Ensemble (10 members, ResNet50) improves FairFace accuracy over individual members in all 6 tasks (e.g., 0.924 vs 0.894 in the main-paper controlled setting; FF age/gender accuracy +0.022 delta), while SPD/EOD/AOD rise significantly for several tasks (e.g., FF age/gender delta SPD +0.022, EOD +
- Deep Ensembles are better calibrated (lower ECE) than individual members on all considered datasets and targets, making them more sensitive to the decision threshold.
- Applying Hardt post-processing to the Deep Ensemble reaches the average fairness violation of individual members without sacrificing accuracy, and maintains performance better than individual members at a 0.05 fairness target.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ensemble members (10-member Deep Ensemble versus individual members) | ResNet50 · FairFace test, age target / gender group | accuracy: .022_{\pm .001} | Helped |
| Add ensemble members (10-member Deep Ensemble versus individual members) | ResNet50 · FairFace test, age target / gender group | SPD: .022_{\pm .003} | Hurt |
| Add ensemble members (10-member Deep Ensemble versus individual members) | ResNet50 · FairFace test, age target / gender group | EOD: .017_{\pm .004} | Hurt |
| Add ensemble members (10-member Deep Ensemble versus individual members) | ResNet50 · FairFace test, age target / gender group | AOD: .017_{\pm .003} | Hurt |
| Add ensemble members (10-member Deep Ensemble versus individual members) | ResNet50 · FairFace test, gender target / age group | EOD: \text{-}.007_{\pm .001} | Helped |
| Add ensemble members (10-member Deep Ensemble versus individual members) | ResNet50 · UTKFace, age target / gender group | accuracy: .015_{\pm .001} | Helped |
| Add ensemble members (10-member Deep Ensemble versus individual members) | ResNet50 · UTKFace, age target / gender group | SPD: .017_{\pm .001} | Hurt |
| Add ensemble members (10-member Deep Ensemble versus individual members) | ResNet50 · CheXpert, age group | AUROC: .005_{\pm .000} | Helped |
| Add ensemble members (10-member Deep Ensemble versus individual members) | ResNet50 · CheXpert, age group | EOD: .008_{\pm .004} | Hurt |
| Set group-dependent thresholds using Hardt post-processing | Deep Ensemble · FairFace, age target / gender group | accuracy | Helped |
| Set group-dependent thresholds using Hardt post-processing | Deep Ensemble · FairFace, age target / gender group | fairness violation | Helped |
| Use non-uniform ensemble-member weights | Deep Ensemble · Considered tasks | performance and fairness trade-off | Mixed |
Surprising findings
- Ensembling can decrease group fairness even though it improves accuracy for every task, contrary to the view that ensembles have only positive impact.
- The largest fairness decrease occurs when the first member is added to the ensemble.
Limitations
- Only vision tasks with convolutional networks are studied.
- Only three group fairness metrics are considered, which cannot guarantee fairness overall.
- Binary protected attributes and binary targets only.
Experimental setup
Datasets: FairFace UTKFace CheXpert FashionMNIST
Benchmarks: FairFace UTKFace CheXpert
Metrics: accuracy AUROC statistical parity difference equal opportunity difference average odds difference expected calibration error balanced accuracy
Baselines: individual ensemble members Hardt post-processing
Builds on: Deep Ensembles Hardt post-processing Ko et al. ensemble subgroup performance analysis
Closest papers by meaning
- Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness (NeurIPS 2025 · similarity 0.90)
- Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness (NeurIPS 2025 · similarity 0.86)
- Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance (NeurIPS 2024 · similarity 0.86)
- FairICP: Encouraging Equalized Odds via Inverse Conditional Permutation (ICML 2025 · similarity 0.85)
- Subgroups Matter for Robust Bias Mitigation (ICML 2025 · similarity 0.85)
- On the Fairness ROAD: Robust Optimization for Adversarial Debiasing (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.