Research map / Image generation / Autoregressive and latent generation
Would Deep Generative Models Amplify Bias in Future Models?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Simulates progressively replacing real COCO and CC3M images with Stable Diffusion-generated images to train OpenCLIP and captioning models, finding that bias does not uniformly amplify and sometimes decreases.
Problem
Whether social biases would be amplified in future vision-language models trained on increasing amounts of AI-generated images from biased deep generative models.
Approach
Simulates dataset contamination by substituting a fraction alpha of real training images with Stable Diffusion v1.5 generations from original captions, then measures how gender, ethnicity, age and skin-tone biases change in OpenCLIP retrieval/self-similarity/person preference and in captioning (LIC, gender error) models.
Key results
- OpenCLIP image retrieval (R@1/5/10) stays roughly constant across contamination levels alpha from 0 to 1 on COCO and Flickr30K.
- Gender bias gap in CC3M text-to-image retrieval and self-similarity decreases with contamination (bias mitigation), while age gap in retrieval widens (amplification).
- ClipCap skin-tone LIC rises from 1.1 at alpha=0 to 3.1 at alpha=1, while Transformer captioning gender LIC and error show no clear trend.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace original COCO training images with Stable Diffusion generations (α=0 to α=1) | ClipCap · COCO 2014 validation | LIC-Skin: 1.1 -> 3.1 | Hurt |
| Replace original COCO training images with Stable Diffusion generations (α=0 to α=1) | ClipCap · COCO 2014 validation | BLEU-4: 31.9 -> 23.8 | Hurt |
| Replace original COCO training images with Stable Diffusion generations (α=0 to α=1) | ClipCap · COCO 2014 validation | CIDEr: 105.0 -> 84.6 | Hurt |
| Replace original COCO training images with Stable Diffusion generations (α=0 to α=1) | ClipCap · COCO 2014 validation | SPICE: 20.4 -> 17.7 | Hurt |
| Replace original COCO training images with Stable Diffusion generations (α=0 to α=1) | ClipCap · COCO 2014 validation | CLIPScore: 76.4 -> 78.3 | Helped |
| Replace original COCO training images with Stable Diffusion generations (α=0 to α=1) | Transformer · COCO 2014 validation | Error: 11.0 -> 13.2 | Hurt |
| Replace original COCO training images with Stable Diffusion generations (α=0 to α=1) | Transformer · COCO 2014 validation | BLEU-4: 28.3 -> 21.0 | Hurt |
| Replace original COCO training images with Stable Diffusion generations (α=0 to α=1) | Transformer · COCO 2014 validation | CIDEr: 92.0 -> 70.3 | Hurt |
| Replace original COCO training images with Stable Diffusion generations (α=0 to α=1) | Transformer · COCO 2014 validation | SPICE: 18.2 -> 14.9 | Hurt |
| Replace original COCO training images with Stable Diffusion generations (α=0 to α=1) | Transformer · COCO 2014 validation | CLIPScore: 72.8 -> 72.9 | No clear effect |
| Increase the proportion of generated CC3M images from α=0.0 to α=1.0 | OpenCLIP · COCO 2014 test set; Flickr30K test set | R@1, R@5, R@10 | No clear effect |
| Increase the proportion of generated CC3M images | OpenCLIP · CC3M validation set (PHASE), age groups | R@5 group gap | Hurt |
Surprising findings
- Introducing generated images does not uniformly amplify bias; several biases decrease, e.g. gender bias gap shrinks as contamination increases.
- Blurry faces generated by Stable Diffusion may hinder learning of demographic attributes, plausibly explaining gender bias mitigation.
Limitations
- Experiments use moderate-scale datasets (about 3M images), far smaller than CLIP or LAION scale.
- Only Stable Diffusion is used for generation, so findings may not transfer to other generators.
- Bias evaluation covers gender, age, ethnicity and skin tone only, excluding intersectional bias.
Experimental setup
Datasets: CC3M COCO 2014 PHASE FairFace Flickr30K
Benchmarks: COCO 2014 1K test Flickr30K test CC3M validation with PHASE annotations COCO validation with gender and skin-tone annotations FairFace
Metrics: R@k self-similarity person preference LIC gender misprediction error BLEU-4 CIDEr SPICE CLIPScore
Baselines: original dataset training (alpha=0)
Builds on: CLIP Stable Diffusion LIC (leakage metric) self-similarity bias metric person preference bias metric
Closest papers by meaning
- Diverse Yet Biased: Towards Mitigating Biases in Generative AI (Student Abstract) (AAAI 2024 · similarity 0.90)
- OpenBias: Open-set Bias Detection in Text-to-Image Generative Models (CVPR 2024 · similarity 0.86)
- Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences (NeurIPS 2024 · similarity 0.86)
- On the Stability of Iterative Retraining of Generative Models on their own Data (ICLR 2024 · similarity 0.86)
- Implicit Bias Injection Attacks against Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.86)
- BIAS-ID: A Framework for Analyzing Transformation Biases in AI-Generated Image Detectors (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.