Research map / Label-efficient and robust learning / Recognition and long-tail
Do Counterfactually Fair Image Classifiers Satisfy Group Fairness? -- A Theoretical and Empirical Study
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds CelebA-CF and LFW-CF counterfactual face benchmarks via IP2P editing with human filtering, shows counterfactual fairness does not imply equalized-odds group fairness on images, and proposes Counterfactual Knowledge Distillation (CKD) to achieve both.
Problem
Whether counterfactual fairness (CF) implies group fairness (GF) in image classification, which was observed on tabular data but is untested on images because real counterfactual images are not available.
Approach
Construct real-image counterfactual benchmarks by editing sex-related facial features with a diffusion editor (InstructPix2Pix) and human filtering; theoretically attribute CF-GF disconnect to a third-party attribute G correlated with the sensitive attribute; propose CKD, distilling a vanilla-trained teacher's robustness to G into a CF-aware student.
Key results
- CP reduces CD on CelebA from 10.26 to 2.53 but raises DEO from 47.10 to 51.01, showing CF does not imply GF on images.
- CKD achieves the best DEO on all three datasets, e.g., CelebA DEO 13.23 vs 47.10 for Scratch, with CD 4.44 vs 10.26.
- Human annotators judged sensitive attribute change accurate at 96.52% (CelebA-CF) and 98.61% (LFW-CF).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CKD to CP training | Image classifiers · CIFAR-10B (α=0.8) | DEO: 33.23 -> 7.30 | Helped |
| Add CKD to CP training | Image classifiers · CelebA | DEO: 51.01 -> 13.23 | Helped |
| Add CKD to CP training | Image classifiers · LFW | DEO: 8.74 -> 1.88 | Helped |
| Add CP regularization to CKD (λ = 0) (ablation) | CKD · CIFAR-10B (α=0.8) | CD: 8.59 -> 2.85 | Helped |
| Add CP regularization to CKD (λ = 0) (ablation) | CKD · CIFAR-10B (α=0.8) | DEO: 11.23 -> 7.30 | Helped |
| Add CP regularization to CKD (λ = 0) (ablation) | CKD · CelebA | CD: 4.31 -> 4.44 | No clear effect |
| Add CP regularization to CKD (λ = 0) (ablation) | CKD · CelebA | DEO: 14.11 -> 13.23 | Helped |
| Add CP regularization to CKD (λ = 0) (ablation) | CKD · LFW | CD: 12.42 -> 7.94 | Helped |
| Add CP regularization to CKD (λ = 0) (ablation) | CKD · LFW | DEO: 2.64 -> 1.88 | Helped |
| Train with CP instead of Scratch | Image classifiers · CelebA | DEO: 47.10 -> 51.01 | Hurt |
Surprising findings
- Counterfactually fair CP models worsen equalized-odds disparity compared with vanilla training, contrary to tabular-data findings that CF implies GF.
- Vanilla-trained models are more robust to the latent attribute G (hair length) than CP-trained models, so they serve as teachers for CKD.
Limitations
- Counterfactual benchmarks are small (230 CelebA and 144 LFW pairs after filtering) and limited to binary sex as sensitive attribute.
- Assumes a vanilla-trained model is robust to G and can serve as teacher; the theory relies on SCM faithfulness assumptions that may not hold, as the paper notes in its appendix.
- Sex as binary label raises ethical concerns the authors acknowledge.
Experimental setup
Datasets: CelebA LFW CIFAR-10
Benchmarks: CelebA-CF LFW-CF CelebA LFW CIFAR-10B
Metrics: accuracy counterfactual disparity disparity of equalized odds rate of flipped predictions
Baselines: Scratch Scratch+aug CP SenSeI LASSI SS RW COV MFD LBC
Builds on: counterfactual pairing Anthis and Veitch CF-GF theorem knowledge distillation InstructPix2Pix
Closest papers by meaning
- FairDD: Fair Dataset Distillation (NeurIPS 2025 · similarity 0.86)
- Fair Deepfake Detectors Can Generalize (NeurIPS 2025 · similarity 0.85)
- FedFACT: A Provable Framework for Controllable Group-Fairness Calibration in Federated Learning (NeurIPS 2025 · similarity 0.85)
- Classes Are Not Equal: An Empirical Study on Image Recognition Fairness (CVPR 2024 · similarity 0.85)
- Counterfactual Fairness by Combining Factual and Counterfactual Predictions (NeurIPS 2024 · similarity 0.85)
- Perceptual Fairness in Image Restoration (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.