Research map / Learning theory and algorithms / Fairness
Who’s the (Multi-)Fairest of Them All: Rethinking Interpolation-Based Data Augmentation Through the Lens of Multicalibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Stress-tests Fair Mixup variants against vanilla Mixup and multicalibration post-processing on folktables employment and income prediction across up to 81 intersecting groups, finding vanilla Mixup (and Mixup combined with MC post-processing) improves multicalibration and balanced accuracy while Fair Mixup generally does not.
Problem
Existing interpolation-based fairness augmentation (Fair Mixup) is evaluated with binary fairness metrics on one minority group, ignoring prediction uncertainty and multiple intersecting groups; multicalibration post-processing wastes scarce minority training data by holding it out.
Approach
Evaluate Mixup and Fair Mixup variants through multicalibration violations, introduce pairwise multicalibration/multiaccuracy fairness penalties and group-balanced batching variants, and combine vanilla Mixup with MC post-processing on a holdout set to get the best fairness-accuracy tradeoff.
Key results
- Mixup+MC enforcement (mixupenforcemc) gives the largest average improvement, up to +14.22% balanced accuracy/MC alpha on Income-Big in the combined metric table, and is best on most settings.
- Fair Mixup worsens baseline performance and fairness on nearly all settings; the only improvement is on Employment-Dis (single disabled group), where FM-DP has the lowest alpha.
- Vanilla Mixup outperforms Fair Mixup and Base on Income in most settings (e.g., +3.12% for Income-er, +3.20% for Income-Dis) while FairBase and FM variants reduce balanced accuracy by up to about 12%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply vanilla Mixup instead of standard training | Base · Employment, multiple group settings | balanced accuracy and worst-group MC α improvement: 2.89 | Helped |
| Apply vanilla Mixup instead of standard training | Base · Income, Dis | balanced accuracy and worst-group MC α improvement: 3.20 | Helped |
| Add MC post-processing after vanilla Mixup | Mixup · Employment All | balanced accuracy and worst-group MC α improvement: 8.31 -> 10.36 | Helped |
| Add MC post-processing after vanilla Mixup | Mixup · Income All | balanced accuracy and worst-group MC α improvement: 9.87 -> 11.64 | Helped |
| Use Fair Mixup for multiple groups rather than vanilla Mixup | Fair Mixup · Most task-setting combinations | worst-group MC α | Hurt |
| Use Fair Mixup for a single disabled group on the binary Employment task | Base · Employment Dis | worst-group MC α | Helped |
| Add interpolated training data (ablation) | FairBase · Multiple group settings | fairness improvement | Helped |
| Balance training batches by minority-group membership (ablation) | Mixup · Multiple group settings | balanced accuracy and fairness | Hurt |
| Add a train-time pairwise fairness penalty (ablation) | Fair Mixup · Multiple group settings | worst-group MC α | Mixed |
| Use MC post-processing without Mixup | Base · Income Dis | balanced accuracy and worst-group MC α improvement: -17.82 | Hurt |
Surprising findings
- Fair Mixup generally worsens both fairness (MC) and balanced accuracy relative to the baseline, except in the single disabled-group Employment setting it was designed for.
- Group-wise data balancing (C1) appears to hurt performance, so the benefit of Mixup comes mainly from learning on interpolated points rather than from balancing.
Limitations
- Only two tabular folktables tasks with a single small MLP; results may not transfer to images, language, or other models.
- Fair Mixup hyperparameters were searched on only one dataset-state-year (California 2022), and a full search was intractable.
- Synthetic interpolation of only one point per pair due to compute constraints.
Experimental setup
Datasets: folktables ACS Employment folktables ACS Income (Adult-style)
Benchmarks: folktables Employment prediction folktables Income prediction
Metrics: balanced accuracy worst-group multicalibration violation alpha percent change over Base
Baselines: Base FairBase Mixup Fair Mixup (demographic parity) Fair Mixup (equalized odds) MC post-processing (enforce MC) MA post-processing (enforce MA)
Builds on: Mixup Fair Mixup multicalibration multiaccuracy boost
Closest papers by meaning
- The Fragility of Fairness: Causal Sensitivity Analysis for Fair Machine Learning (NeurIPS 2024 · similarity 0.88)
- Multiaccuracy and Multicalibration via Proxy Groups (ICML 2025 · similarity 0.87)
- Fair Risk Control: A Generalized Framework for Calibrating Multi-group Fairness Risks (ICML 2024 · similarity 0.87)
- FairWASP: Fast and Optimal Fair Wasserstein Pre-processing (AAAI 2024 · similarity 0.87)
- Achievable Fairness on Your Data With Utility Guarantees (NeurIPS 2024 · similarity 0.87)
- On the Burden of Achieving Fairness in Conformal Prediction (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.