Research map / Learning theory and algorithms / Fairness
Rethinking Fair Representation Learning for Performance-Sensitive Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically that fair representation learning cannot be both effective and harmless on IID data, and empirically on medical imaging under distribution shift that FRL's benefit depends on bias mechanism and subgroup separability.
Problem
Whether fair representation learning (FRL), designed for demographic parity, is valid for performance-sensitive tasks, i.e., maximising subgroup-wise accuracy under IID or distribution-shift evaluation.
Approach
Uses a causal SCM view of dataset bias (presentation, prevalence, annotation disparities) to prove that FRL cannot be both effective and harmless when train and test distributions coincide, and hypothesises that FRL validity under shift depends on bias structure and subgroup separability.
Key results
- Proposition shows FRL cannot be both effective and harmless on IID train/test data; empirically ΔAcc is negative or near zero for FRL vs ERM across IID settings.
- Under distribution shift, FRL only improves over ERM for presentation disparities with high subgroup separability, where it boosts the disadvantaged group.
- Aggregated over bias mechanisms, ΔAcc shows a statistically significant monotonic association with subgroup separability (Kendall's tau).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply marginal fair representation learning (FRL) instead of ERM | Medical disease classification · IID, unbiased train/test datasets | subgroup accuracy | Mixed |
| Apply conditional FRL instead of ERM | Medical disease classification · IID, unbiased train/test datasets | subgroup accuracy | Mixed |
| Apply FRL instead of ERM when training data has presentation disparities | Medical disease classification · Unbiased test set; high subgroup separability | Group 1 accuracy | Helped |
| Apply FRL instead of ERM when training data has presentation disparities | Medical disease classification · Unbiased test set; high subgroup separability | Group 0 accuracy | No clear effect |
| Apply FRL instead of ERM when training data has prevalence disparities | Medical disease classification · Unbiased test set | subgroup accuracy | No clear effect |
| Apply FRL instead of ERM when training data has annotation disparities | Medical disease classification · Unbiased test set | subgroup accuracy | No clear effect |
| Apply FRL instead of ERM on data with low subgroup separability | Medical disease classification · Unbiased test set; all bias mechanisms | subgroup accuracy | Hurt |
| Apply FRL instead of ERM on data with high subgroup separability | Medical disease classification · Unbiased test set; presentation disparities | Group 1 accuracy | Helped |
| Increase subgroup separability | FRL relative to ERM · Unbiased test set; results aggregated over bias mechanisms | accuracy gap (Δ Acc) | Helped |
Surprising findings
- FRL degrades performance for both groups when subgroup separability is low, even on unbiased IID data.
- Dataset-attribute combinations with the lowest subgroup separability suffered the largest FRL penalty.
Limitations
- Theoretical results do not bound the size of the performance penalty.
- Biased datasets are synthetically generated, which may not capture real-world bias complexity.
- Only one adversarial and one conditional FRL method evaluated, with binary sensitive attributes.
Experimental setup
Datasets: chest X-ray dataset dermatoscopy dataset fundus imaging dataset (PAPILA)
Benchmarks: chest X-ray, dermatoscopy, fundus disease classification (biased/unbiased variants)
Metrics: subgroup-wise accuracy difference (percentage points) AUC Kendall's tau
Baselines: ERM conditional FRL
Builds on: fair representation learning counterfactual fairness information bottleneck d-separation
Closest papers by meaning
- Causal Fairness under Unobserved Confounding: A Neural Sensitivity Framework (ICLR 2024 · similarity 0.88)
- Kernel-based Equalized Odds: A Quantification of Accuracy-Fairness Trade-off in Fair Representation Learning (NeurIPS 2025 · similarity 0.88)
- The Fragility of Fairness: Causal Sensitivity Analysis for Fair Machine Learning (NeurIPS 2024 · similarity 0.87)
- FairPFN: A Tabular Foundation Model for Causal Fairness (ICML 2025 · similarity 0.87)
- The Boundaries of Fair AI in Medical Image Prognosis: A Causal Perspective (NeurIPS 2025 · similarity 0.87)
- FairCLIP: Harnessing Fairness in Vision-Language Learning (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.