Research map / Vision-language and multimodal / Multimodal LLMs
BBQ-V: Benchmarking Visual Stereotype Bias in Large Multimodal Models
Vishal Narnaware, Ashmal Vayani, Rohit Gupta, Swetha Sirnam, Shah Mubarak
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add image input (vision eval vs blind eval) (ablation) | Gemma-3-12B · BBQ-V | fairness score: 80.06% -> 63.41% | Hurt |
| add image input (vision eval vs blind eval) (ablation) | LLaMA-3.2-Vision-11B · BBQ-V | fairness score: 62.80% -> 43.36% | Hurt |
| add image input (vision eval vs blind eval) (ablation) | Qwen2.5-VL-7B · BBQ-V | fairness score: 81.19% -> 65.53% | Hurt |
| add image input (vision eval vs blind eval) (ablation) | Phi-4-MM-Instruct · BBQ-V | fairness score: 89.44% -> 74.30% | Hurt |
| use thinking/reasoning model | GLM-4.1V-9B-Thinking · BBQ-V | fairness score: 53.58% overall | Hurt |
| use thinking/reasoning model | Qwen3-VL-8B-Thinking · BBQ-V | fairness score: 84.26% Socio-Economic vs 52.04% Physical Appearance | Mixed |
| scale up from GPT-4o-mini to GPT-4o | GPT-4o family · BBQ-V | fairness score: 61.53% -> 75.38% | Improved |
| use proprietary Gemini-2.5-flash-lite over open-source models | Gemini-2.5-flash-lite · BBQ-V | fairness score: 80.89% vs 74.30% best open-source | Improved |
Closest papers by meaning
- OASIS Uncovers: High-Quality T2I Models, Same Old Stereotypes (ICLR 2025 · similarity 0.90)
- SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples (CVPR 2024 · similarity 0.88)
- Discovering and Mitigating Visual Biases through Keyword Explanation (CVPR 2024 · similarity 0.87)
- On Fairness of Unified Multimodal Large Language Model for Image Generation (NeurIPS 2025 · similarity 0.87)
- Interpretable Debiasing of Vision-Language Models for Social Fairness (CVPR 2026 · similarity 0.87)
- StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
