Research map / Image generation / Text-to-image diffusion
Association of Objects May Engender Stereotypes: Mitigating Association-Engendered Stereotypes in Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAS, a framework that mitigates association-engendered stereotypes in text-to-image diffusion models by aligning generated stereotype distributions with a stereotype-free distribution via PIS CLIP and a Sensitive Transformer, and introduces the SDTV metric.
Problem
Text-to-image models produce stereotypes when multiple objects are associated in a prompt (e.g., houses linked to race), which existing mitigation methods that address single objects fail to handle.
Approach
Models stereotype mitigation as probability distribution alignment: a CLIP-style PIS model learns prompt-image-stereotype mappings, a Sensitive Transformer produces sensitive constraints embedded into the diffusion process, and a new Total-Variation-based metric (SDTV) measures stereotype severity.
Key results
- On SD-1.5, MAS reduces M-O & S-SA gender SDTV from .86 to .17-ish range and M-O & M-SA G.xR. from .75 to .21 (table reports MAS values .17 to .23 across categories versus .47-.96 originals).
- Compared with six prior mitigation methods, MAS achieves the lowest SDTV across the eight single- and multi-object scenarios in Table 3 on SD-1.5.
- MAS keeps FID and CLIP-T2I close to the original models (e.g., SD-1.5 CLIP-T2I .38 vs .39 original), with only small generation-time overhead.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MAS to SD-1.5 | SD-1.5 · M-O & S-SA, Gender | SDTV: .57±.21 -> .17±.11 | Helped |
| Apply MAS to SD-1.5 | SD-1.5 · M-O & S-SA, Race | SDTV: .49±.16 -> .20±.09 | Helped |
| Apply MAS to SD-1.5 | SD-1.5 · M-O & S-SA, Region | SDTV: .56±.11 -> .20±.02 | Helped |
| Apply MAS to SD-1.5 | SD-1.5 · M-O & M-SA, Gender × Race | SDTV: .47±.23 -> .16±.10 | Helped |
| Apply MAS to SD-1.5 | SD-1.5 · S-O & S-SA, Gender | SDTV: .68±.27 -> .17±.14 | Helped |
| Apply MAS to SD-1.5 | SD-1.5 · S-O & S-SA, Race | SDTV: .82±.14 -> .21±.09 | Helped |
| Apply MAS to SD-1.5 | SD-1.5 · S-O & S-SA, Region | SDTV: .81±.10 -> .23±.13 | Helped |
| Apply MAS to SD-1.5 | SD-1.5 · S-O & M-SA, Gender × Race | SDTV: .75±.20 -> .21±.02 | Helped |
Surprising findings
- Newer distilled models (Lightning, Turbo, Cascade) display more pronounced stereotypes than SD-1.5 and SD XL.
- Prior mitigation methods (Kim 2023, Chuang 2023, etc.) cannot address several multi-object categories at all, reported as '-'.
Limitations
- Only addresses the stereotype types covered by the chosen templates and sensitive attributes (gender, race, region, religion, adjectives); subtle stereotypes left for future work.
- Evaluation relies on a fixed set of 10 prompts per category and sensitive attribute classifiers, limiting coverage.
- Stereotype-free target distribution assumed to be uniform across attribute values, which may not match all notions of fairness.
Experimental setup
Datasets: COCO FairFace LAION-5B Stereoset DiffusionDB
Benchmarks: SDTV on template prompts across four stereotype categories DiffusionDB non-template prompts FID and generation time CLIP score semantic preservation
Metrics: SDTV FID CLIP-T2I CLIP-I2I generation time
Baselines: Kim 2023 Chuang 2023 Gandikota 2024 Bansal 2022 Wang 2023 Shen 2024
Builds on: CLIP Transformer attention Safe Latent Diffusion
Closest papers by meaning
- OASIS Uncovers: High-Quality T2I Models, Same Old Stereotypes (ICLR 2025 · similarity 0.89)
- Implicit Bias Injection Attacks against Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.87)
- FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models (NeurIPS 2025 · similarity 0.86)
- Multi-Group Proportional Representations for Text-to-Image Models (CVPR 2025 · similarity 0.86)
- SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples (CVPR 2024 · similarity 0.86)
- Discovering and Mitigating Visual Biases through Keyword Explanation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.