Research map / Vision-language and multimodal / Human and embodied modeling
Understanding and Evaluating Racial Biases in Image Captioning
Dora Zhao, Angelina Wang, Olga Russakovsky
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add discriminability loss (DiscCap vs Att2in) (ablation) | Att2in · COCO 2014 val | CIDEr gap lighter-darker: 2.7 ± 0.7 -> 0.6 ± 0.7 | Mixed |
| use newer transformer-based models instead of FC | FC · COCO 2014 val | CIDEr gap lighter-darker: -0.8 ± 1.8 -> 3.2 ± 1.2 | Hurt |
| compare SPICE across skin tone groups | all six captioners · COCO 2014 val | SPICE gap: 0.0 ± 0.1 | No clear effect |
| switch from human to generated captions (Transformer) | Human captions · Controlled pairs | embedding classifier AUC: 68.9 -> 54.2 | Improved |
| switch from FC to AoANet captions | FC · Controlled pairs | vocabulary classifier AUC: 65.9 -> 68.0 | Hurt |
| switch from FC to Oscar (larger pretraining corpus) | FC · Controlled pairs | vocabulary classifier AUC: 65.9 -> 64.4 | Improved |
| switch from human to generated captions | Human captions · Controlled pairs | VADER sentiment gap: 1.5 -> 0.3 | Improved |
| add gender feature to distance-from-center | logistic regression on layout · Skin tone classification | AUC: 56.6 -> 57.8 | Improved |
| match images by similarity and mask people (controlled set) | visual classifier · Skin tone classification | AUC: 55.4 -> 44.7 | Improved |
| lighter versus darker skin images in human captions | Human captions · COCO sentiment | VADER compound: 0.073 -> 0.059 | Hurt |
| racial descriptors for individuals who appear Black versus white | Human captions · COCO train captions | descriptor rate: 0.03% -> 0.54% | Hurt |
Closest papers by meaning
- SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples (CVPR 2024 · similarity 0.88)
- FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models (NeurIPS 2025 · similarity 0.86)
- Finetuning Text-to-Image Diffusion Models for Fairness (ICLR 2024 · similarity 0.86)
- Discovering and Mitigating Visual Biases through Keyword Explanation (CVPR 2024 · similarity 0.86)
- Association of Objects May Engender Stereotypes: Mitigating Association-Engendered Stereotypes in Text-to-Image Generation (NeurIPS 2024 · similarity 0.85)
- Implicit Bias Injection Attacks against Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
