Research map / Optimization / Optimizers
Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses an SDE analysis of a toy two-group setup plus two theorems to argue that RMSProp (and Adam) yield fairer minima and updates than SGD, validated on CelebA, FairFace and MS-COCO with ViT and CNN backbones.
Problem
Whether and how the choice of optimization algorithm (adaptive vs. stochastic gradient methods) affects group fairness in deep neural networks, especially under data imbalance.
Approach
Models SGD and RMSProp as SDEs on a two-subgroup toy problem with Fokker-Planck analysis showing RMSProp is more likely to converge near the fair minimum under severe sampling bias; adds theorems that RMSProp's per-coordinate scaling shrinks subgroup update disparities and bounds the one-step demographic parity gap no worse than SGD.
Key results
- On CelebA with gender as sensitive attribute, Adam improves F_EOD over SGD by about 8%, and RMSProp outperforms SGD in 4 of 5 F_EOD scenarios.
- Wilcoxon signed-rank tests over 10 CelebA runs give p-values mostly on the order of 1e-3 for SGD vs. RMSProp and SGD vs. Adam on fairness metrics.
- In the toy SDE simulation with p0=0.1, roughly 10% of RMSProp runs fall within the fair neighborhood of w=0 while SGD rarely does.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Adam instead of SGD | ViT · CelebA, gender | F_EOD: 62.66 -> 65.21 | Helped |
| Use RMSProp instead of SGD | ViT · CelebA, gender | F_EOD: 62.66 -> 65.18 | Helped |
| Use Adam instead of SGD | ViT · CelebA | accuracy: 91.23 -> 92.08 | Helped |
| Use RMSProp instead of SGD | ViT · CelebA | F1: 92.12 -> 91.17 | Hurt |
| Use Adam instead of SGD | ViT · MS-COCO | F1: 68.35 -> 74.10 | Helped |
| Use RMSProp instead of SGD | ViT · FairFace | accuracy: 89.41 -> 91.37 | Helped |
| Use Adam instead of SGD | ViT · FairFace | F1: 91.13 -> 92.17 | Helped |
| Use Adam instead of SGD | models trained with the fairness-enhancing method · ProPublica COMPAS | gap in Demographic Parity: 2.60 -> 0.86 | Helped |
| Use Adam instead of SGD | models trained with the fairness-enhancing method · AdultCensus | gap in Equalized Odds: 4.28 -> 2.39 | Helped |
Surprising findings
- Adaptive optimizers improve fairness even when accuracy is comparable, so the fairness gain is not explained by overall performance.
- Fairness gap between RMSProp and SGD shrinks as the dataset becomes more balanced in the CelebA downsampling experiment.
Limitations
- Theoretical guarantees rely on toy one-dimensional setups, isotropic or anisotropic Gaussian noise assumptions, and approximations (e.g., variance of v_k negligible).
- Fairness improvements are modest in several scenarios and the MS-COCO F_EOD values are zero for both optimizers due to rare classes.
Experimental setup
Datasets: CelebA FairFace MS-COCO ProPublica COMPAS AdultCensus
Benchmarks: CelebA FairFace MS-COCO ProPublica COMPAS AdultCensus
Metrics: equalized odds (F_EOD) equal opportunity (F_EOP) demographic parity (F_DPA) accuracy F1 score
Baselines: SGD Adam SGD with momentum AdamW AdaBound
Builds on: RMSProp Adam SGD focal loss Hyperband
Closest papers by meaning
- The Disparate Benefits of Deep Ensembles (ICML 2025 · similarity 0.90)
- On the Fairness ROAD: Robust Optimization for Adversarial Debiasing (ICLR 2024 · similarity 0.87)
- FairICP: Encouraging Equalized Odds via Inverse Conditional Permutation (ICML 2025 · similarity 0.87)
- Enhancing Group Fairness in Online Settings Using Oblique Decision Forests (ICLR 2024 · similarity 0.87)
- Stay Fair! Ensuring Group Fairness in Diffusion Models Across Guidance Scales (NeurIPS 2026 · similarity 0.86)
- FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.