Research map / Vision-language and multimodal / Human and embodied modeling
Bias in Gender Bias Benchmarks: How Spurious Features Distort Evaluation
Yusuke Hirota, Ryo Hachiuma, Boyi Li, Ximing Lu, Michael Ross Boone, Boris Ivanovic, Yejin Choi, Marco Pavone, Yu-Chiang Frank Wang, Noa Garcia, Yuta Nakashima, Chao-Han Huck Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| perturb background by blurring (weak) | generative VLMs bias evaluation · COCO-gender YGap | relative difference Delta: 165.84% for LLaVA-OneVision-7B | Hurt |
| mask 10% of objects | generative VLMs bias evaluation · FACET / MIAP YGap | relative difference Delta: 40.98% (Qwen2-VL-7B, FACET), 48.86% (InternVL-2.5-8B, MIAP) | Hurt |
| perturb lighting (brightness) | bias evaluation · all four benchmarks | relative difference Delta: mostly below 10% for generative VLMs, below 2% for CLIP | No clear effect |
| shift hue (color perturbation) | bias evaluation · MIAP YGap | relative difference Delta: 20.88% (EAGLE-8B, strong) | Mixed |
| perturb object and background features for CLIP variants | CLIP MaxSkew@1000 · FACET / PHASE | relative difference Delta: 25.03% object (ViT-B/32 FACET), 51.01% background (ViT-B/32 PHASE) | Hurt |
| stronger spurious correlation of a feature with gender | bias evaluation · generative VLMs vs CLIP variants | correlation of Acc_b with Delta: r = 0.31 (generative), r = 0.54 (CLIP) | Improved |
| perturb object features | model bias rankings · FACET YGap ranking | bias ranking | Hurt |
| use ResNet-50 instead of ConvNeXt-B for feature gender classifier (ablation) | spurious feature detection · four benchmarks | gender prediction accuracy: COCO-gender object 76.3 -> 76.3 | No clear effect |
Closest papers by meaning
- Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs) (ICLR 2025 · similarity 0.90)
- SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples (CVPR 2024 · similarity 0.87)
- A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks (NeurIPS 2024 · similarity 0.86)
- Joint Vision-Language Social Bias Removal for CLIP (CVPR 2025 · similarity 0.86)
- Are Models Biased on Text without Gender-related Language? (ICLR 2024 · similarity 0.86)
- On Fairness of Unified Multimodal Large Language Model for Image Generation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
