Research map / Image generation / Text-to-image diffusion
Fair Text-to-Image Diffusion via Fair Mapping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Fair Mapping, a lightweight 8-layer linear network inserted after the text encoder of Stable Diffusion 1.5 (frozen diffusion model) trained with a semantic-consistency loss and a distance-variance fairness loss to reduce gender and race bias in human-related prompts.
Problem
Text-to-image diffusion models generate demographically biased images for human-related prompts (e.g., occupations and emotions) even when no sensitive attribute is specified, because implicit language bias in text embeddings propagates into generation.
Approach
A model-agnostic linear mapping network after the frozen text encoder is trained to keep embeddings semantically consistent while equalizing their distances to sensitive-attribute prompt embeddings; a prompt detector decides whether to route inputs through the mapping at inference.
Key results
- Fair Mapping reduces gender Diffusion Bias from 0.4466 (SD) to 0.3625 on occupations and 0.4652 to 0.2113 on emotions, the lowest among compared methods.
- Race Diffusion Bias drops from 0.2599 to 0.2231 (occupation) and 0.1893 to 0.1178 (emotion) relative to SD.
- Generating 100 images takes 434 seconds versus 424 for SD, while training on 150 occupations takes about 50 minutes on one V100; CLIP-Score falls slightly (0.2320 to 0.2021 for gender occupation).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Fair Mapping | Stable Diffusion 1.5 · Occupation, gender | Bias: 0.4466 -> 0.3625 | Helped |
| Apply Fair Mapping | Stable Diffusion 1.5 · Emotion, gender | Bias: 0.4652 -> 0.2113 | Helped |
| Apply Fair Mapping | Stable Diffusion 1.5 · Occupation, race | Bias: 0.2599 -> 0.2231 | Helped |
| Apply Fair Mapping | Stable Diffusion 1.5 · Emotion, race | Bias: 0.1893 -> 0.1178 | Helped |
| Apply Fair Mapping for gender | Stable Diffusion 1.5 · Occupation | CLIP-Human: 0.1339 -> 0.1494 | Helped |
| Apply Fair Mapping for gender | Stable Diffusion 1.5 · Occupation | CLIP-Score: 0.2320 -> 0.2021 | Hurt |
| Apply Fair Mapping for gender | Stable Diffusion 1.5 · Occupation | ICAD: 3.64 to 3.69 | Helped |
| Remove L_fair (ablation) | Fair Mapping with L_text and L_fair · Occupation, gender | Bias: 0.3624 -> 0.4030 | Hurt |
| Remove L_fair (ablation) | Fair Mapping with L_text and L_fair · Emotion, gender | Bias: 0.2113 -> 0.3862 | Hurt |
| Use L_fair alone without L_text (ablation) | Fair Mapping · Occupation, gender | Bias | No clear effect |
| Increase lambda above 0.1 (ablation) | Fair Mapping · Occupation and Emotion | BiasScore | Hurt |
| Increase lambda (ablation) | Fair Mapping · Occupation and Emotion | CLIP-Score | Hurt |
Surprising findings
- Language bias (distance of biased-attribute prompt embeddings from neutral prompt embeddings) correlates with diffusion bias, suggesting the two are mutually reinforcing.
- Increasing the fairness penalty lambda beyond 0.1 raises BiasScore again due to image distortion, as the authors attribute it to loss of semantic information.
Limitations
- Fair Mapping introduces a trade-off with prompt-image alignment, especially facial expression consistency per human study.
- Fairness evaluated only for gender and race on occupation and emotion keywords with face-generation focus.
- Quality metrics show a slight CLIP-Score drop and some human-preference alignment loss for emotions.
Experimental setup
Datasets: LAION-5B (base model pretraining, not used directly) occupation and emotion keyword sets
Benchmarks: occupation keyword set (150 occupations) emotion keyword set (20 emotions)
Metrics: Diffusion Bias (BiasScore) CLIP-Score Human-CLIP ICAD human preference fidelity and alignment generation time
Baselines: Stable Diffusion Structured Diffusion Composable Diffusion Fair Diffusion Unified Concept Editing debiasVL
Builds on: Stable Diffusion classifier-free guidance DDIM MixFairFace
Closest papers by meaning
- FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models (NeurIPS 2025 · similarity 0.93)
- Finetuning Text-to-Image Diffusion Models for Fairness (ICLR 2024 · similarity 0.91)
- Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusion (CVPR 2025 · similarity 0.90)
- FairRAG: Fair Human Generation via Fair Retrieval Augmentation (CVPR 2024 · similarity 0.90)
- LightFair: Towards an Efficient Alternative for Fair T2I Diffusion via Debiasing Pre-trained Text Encoders (NeurIPS 2025 · similarity 0.88)
- FairQueue: Rethinking Prompt Learning for Fair Text-to-Image Generation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.