Research map / Image generation / Text-to-image diffusion
Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that Stable Diffusion already contains "minority regions" in noise space and proposes weak text-embedding guidance that steers random noise toward them to reduce gender and racial bias without any training.
Problem
Text-to-image models such as Stable Diffusion exhibit demographic bias (e.g., gender and race skew in professions), and existing de-biasing methods require costly additional training that can degrade image generation quality.
Approach
Introduces a "mode test" showing that initial noises associated with minority attributes cluster into regions, then de-biases by adding a scaled attribute-embedding direction only at positions from the [EOS] token onward in the text embedding for the first denoising steps, alternating with the original embedding afterward.
Key results
- On SD1.5 over eight professions, the method reaches average deviation 0.167 from the 0.5 target versus 0.403 for vanilla SD, matching FairDiffusion while keeping CLIP score (26.56) and CMMD (0.509) at vanilla level.
- CMMD on MS-COCO for FairDiffusion rises to 0.586 and UCE to 1.240 on SD1.5, whereas the proposed method stays near vanilla SD (0.532).
- Average deviation reduced across SD versions: SD2 0.444 to 0.362, SDXL 0.363 to 0.242, SD3 0.481 to 0.349.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply weak attribute guidance | Stable Diffusion 1.5 · Eight professions | Avg. ∆ from target minor-attribute ratio: 0.403 -> 0.167 | Helped |
| Apply weak attribute guidance | Stable Diffusion 2 · Eight professions | Avg. ∆ from target minor-attribute ratio: 0.444 -> 0.362 | Helped |
| Apply weak attribute guidance | Stable Diffusion XL · Eight professions | Avg. ∆ from target minor-attribute ratio: 0.363 -> 0.242 | Helped |
| Apply weak attribute guidance | Stable Diffusion 3 · Eight professions | Avg. ∆ from target minor-attribute ratio: 0.481 -> 0.349 | Helped |
| Add weak attribute guidance | Stable Diffusion 1.5 · MS COCO 2014-30k | CLIP Score: 26.42 -> 26.56 | Helped |
| Add weak attribute guidance | Stable Diffusion 2 · MS COCO 2014-30k | CMMD: 0.524 -> 0.549 | Hurt |
| Add weak attribute guidance | Stable Diffusion 3 · MS COCO 2014-30k | CMMD: 0.646 -> 0.664 | Hurt |
| Reduce CFG scale (ablation) | Stable Diffusion · Gender bias across professions | Major-attribute ratio | Helped |
| Reduce CFG scale (ablation) | Stable Diffusion · Gender bias across professions | CLIP score | Hurt |
| Inject more noise into the text condition (ablation) | CADS · Gender and racial bias | Major-attribute ratio | Helped |
| Inject noise into the text condition (ablation) | CADS · Gender and racial bias | CLIP score | Hurt |
| Use minor-attribute prompts for a larger fraction of diffusion steps (ablation) | Stable Diffusion · Gender bias across professions | Minor-attribute ratio | Helped |
Surprising findings
- Training-based de-biasing methods (UCE, FTDiff, SelfDisc) often fail to reach balanced target ratios even with additional training.
- Adding minority-attribute guidance at every token position rather than only from [EOS] causes the model to ignore explicit gender in prompts, showing that excessive guidance overrides user intent.
Limitations
- Only gender and race biases are studied, with binary gender and a fixed set of professions.
- Bias evaluation relies on CLIP zero-shot classifiers, which may be inaccurate.
- Some attribute-specific prompt configurations reduce CLIP alignment for the CADS-style noise analysis, though the final method preserves it.
Experimental setup
Datasets: LAION-5B MS-COCO 2014
Benchmarks: MS-COCO 2014 30k
Metrics: CLIP score CMMD minor attribute ratio average absolute deviation from 0.5
Baselines: FairDiffusion Unified Concept Editing (UCE) Fine-Tuning Diffusion (FTDiff) Self-Discovering latent direction (SelfDisc)
Builds on: Classifier-Free Guidance Condition-Annealed Sampling (CADS) SDEdit FairDiffusion
Closest papers by meaning
- Minority-Focused Text-to-Image Generation via Prompt Optimization (CVPR 2025 · similarity 0.90)
- FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models (NeurIPS 2025 · similarity 0.90)
- Fair Text-to-Image Diffusion via Fair Mapping (AAAI 2025 · similarity 0.90)
- InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization (CVPR 2024 · similarity 0.90)
- LightFair: Towards an Efficient Alternative for Fair T2I Diffusion via Debiasing Pre-trained Text Encoders (NeurIPS 2025 · similarity 0.89)
- OneActor: Consistent Subject Generation via Cluster-Conditioned Guidance (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.