Research map / Image generation / Text-to-image diffusion
AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes a real-data-biased ResNet-18 on balanced synthetic images from Stable Diffusion v2.1 (prompted by GPT-4 contextual prompts), updating only parameters selected as fairness-sensitive and domain-insensitive via gradient-difference rankings, improving fairness without demographic labels.
Problem
Biased facial attribute classifiers trained on imbalanced real data show poor worst-group accuracy and equalized odds; prior synthetic-data and generative-editing fixes either lose accuracy, need group annotations, or suffer from low-quality synthetic data and domain/bias shift between real and synt
Approach
Generate diverse balanced synthetic data with a text-to-image LDM conditioned on LLM-written contextual prompts, then selectively fine-tune only parameters whose gradient differences show high fairness sensitivity and low domain sensitivity, using a mask applied during fine-tuning.
Key results
- On CelebA (Smiling, Male), AIM-Fair reports ACC 89.02, WST 84.20, EO 6.07, versus DiGA at 88.40, 81.90, 7.40.
- On CelebA (Smiling, Young), AIM-Fair reaches WST 87.78 and EO 5.44 versus DiGA's 78.50 and 9.50.
- Under 10% real-data subsampling on CelebA, AIM-Fair attains WST 81.73 and EO 8.16, outperforming all baselines except DiGA on ACC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AIM-Fair instead of DiGA | ResNet-18 · CelebA (Smiling; protected Male) | WST: 81.90 -> 84.20 | Helped |
| Use AIM-Fair instead of DiGA | ResNet-18 · CelebA (Smiling; protected Young) | WST: 78.50 -> 87.78 | Helped |
| Use AIM-Fair instead of DiGA | ResNet-18 · CelebA (Young; protected Male) | ACC: 80.00 -> 78.19 | Hurt |
| Replace full fine-tuning with selective fine-tuning (ablation) | ResNet-18 · UTKFace (Female; protected White) | WST: 83.00 -> 84.26 | Helped |
| Generate training images with contextual prompts rather than a plain prompt (ablation) | Synthetic-data classifier · CelebA (Smiling; protected Male) | WST: 43.28 -> 78.52 | Helped |
| Increase contextual prompt count from 25 to 50 (ablation) | Synthetic-data classifier · CelebA (Smiling; protected Male) | WST: 69.73 -> 72.22 | Helped |
| Add head-pose variation to 50 contextual prompts (ablation) | Synthetic-data classifier · CelebA (Smiling; protected Male) | WST: 72.22 -> 78.52 | Helped |
| Use absolute gradient differences instead of cosine similarity for parameter selection (ablation) | Selective fine-tuning · CelebA (Smiling; protected Male) | WST: 83.33 -> 84.20 | Helped |
| Set top-k parameter selection to 55 (ablation) | AIM-Fair · CelebA (Smiling; protected Male) | Fairness | Helped |
| Match biased synthetic-data bias ratio to the real-data ratio (1:9) (ablation) | AIM-Fair · CelebA (Smiling; protected Male) | WST: 83.90 -> 84.20 | Helped |
| Use synthetic data equal in amount to the real training data (ablation) | AIM-Fair · CelebA (Smiling; protected Male) | WST: 80.80 -> 84.20 | Helped |
| Use half as much synthetic data as real training data (ablation) | AIM-Fair · CelebA (Smiling; protected Male) | ACC: 89.02 -> 89.51 | Helped |
Surprising findings
- Full fine-tuning on synthetic data reduces utility, while freezing one block tends to improve fairness at the cost of accuracy.
- Cosine-similarity-based parameter selection performed worse than absolute gradient differences.
Limitations
- Relies on a pre-trained biased model and a hyperparameter top-k and bias ratio for synthetic data, with performance sensitive to these choices.
- Evaluated only on two face datasets with binary protected attributes and ResNet-18.
- Accuracy on some settings (e.g., Young/Male on CelebA) trails DiGA.
Experimental setup
Datasets: CelebA UTKFace
Benchmarks: CelebA UTKFace
Metrics: overall accuracy worst-group accuracy equalized odds group standard deviation
Baselines: ERM CVaR DRO EIIL LfF JTT MAPLE DiGA Linear Probing Fully Fine-Tuning Sub-Tuning Data Supplementation Data Repairing
Builds on: Stable Diffusion DiGA Sub-Tuning Just Train Twice
Closest papers by meaning
- Stay Fair! Ensuring Group Fairness in Diffusion Models Across Guidance Scales (NeurIPS 2026 · similarity 0.89)
- Finetuning Text-to-Image Diffusion Models for Fairness (ICLR 2024 · similarity 0.88)
- On Optimal Steering to Achieve Exact Fairness (NeurIPS 2025 · similarity 0.88)
- FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models (NeurIPS 2025 · similarity 0.87)
- Diverse Yet Biased: Towards Mitigating Biases in Generative AI (Student Abstract) (AAAI 2024 · similarity 0.87)
- Intersectional Unfairness Discovery (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.