Research map / Vision-language and multimodal / Multimodal LLMs
Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models
Yusuke Hirota, Michael Boone, Arun Zachariah, Jibin Rajan Varghese, Yu-Chiang Frank Wang, Boyi Li, Ryo Hachiuma
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace attribute-inferring prompts with person-irrelevant prompts plus user photo | GPT-5 · bias benchmarks | refusal rate: 83 (SBBench) -> 0 | Improved |
| replace attribute-inferring prompts with person-irrelevant prompts plus user photo | Claude 3.7 Sonnet · bias benchmarks | refusal rate: 100 (SBBench) -> 0 | Improved |
| treat image as provisional user information instead of captioning target (ablation) | InternVL3-14B · vocabulary shift across image domains | JS distance: 0.61 -> 0.38 | Improved |
| use proprietary models instead of open-source | LVLMs · Story generation | mean bias (gender, race): 29.29 -> 18.99 | Improved |
| use proprietary models instead of open-source | LVLMs · Term explanation | mean bias (gender, race): 4.71 -> 4.49 | No clear effect |
| make task more constrained (story to term to exam QA) | LVLMs · three tasks | bias score: 27.23 -> 4.67 -> 1.48 | Improved |
| increase model size | open-source LVLMs · Story generation | racial bias: r = 0.72 | Hurt |
| increase model performance (MMMU) | LVLMs · Exam-style QA | bias score: r = -0.81 / -0.84 | Improved |
| safety-aware training (Gemma3) | Gemma3 · bias scores | bias score | No clear effect |
| LLM assistant vs human judgment | Qwen3-32B · Term explanation | agreement: 97 of 100 | Improved |
Closest papers by meaning
- SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models (AAAI 2024 · similarity 0.88)
- SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP (ICLR 2025 · similarity 0.87)
- SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples (CVPR 2024 · similarity 0.87)
- Vision Language Models are Biased (ICLR 2026 · similarity 0.87)
- More of the Same: Persistent Representational Harms Under Increased Representation (NeurIPS 2025 · similarity 0.87)
- Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs) (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
