Research map / Vision-language and multimodal / Multimodal LLMs
See It from My Perspective: How Language Affects Cultural Bias in Image Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows VLMs exhibit a Western bias in culturally diverse object identification, VQA and art emotion tasks, and that a Chinese-heavy LLM pre-training mix (Baichuan2 vs Llama2 in LLaVA-style models) reduces this bias, more than prompting language alone.
Problem
Vision-language models may reflect a Western cultural perspective in image understanding, and it is unclear how the language mix used in pre-training and prompting affects this bias in objective and subjective visual tasks.
Approach
Defines a Western-vs-East-Asian error-disparity bias metric across tasks, then trains controlled LLaVA-style mLLaVA variants that vary the base LLM's pre-training language mix (Llama2 vs Baichuan2), prompting language, and fusion corpus language to isolate language effects on bias.
Key results
- 90%+ of off-the-shelf VLMs show Western bias on every task, but Chinese prompting reduces bias in only 15/30 model-task evaluations.
- Switching the base LLM from mostly English Llama2 to bilingual Baichuan2 reduces ArtELingo (opposite valence) Western bias from 2.47 to 1.23 when prompting in Chinese.
- More Chinese in LLM pre-training reduces subjective-task bias even when prompting in English (ArtELingo opposite valence 3.45 to 2.71).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Baichuan2 instead of Llama2 as the base LLM, adding more Chinese to text-only pre-training | mLLaVA prompted in Chinese · VQA en/zh | Western bias: 1.50 to 1.17 | Helped |
| Use Baichuan2 instead of Llama2 as the base LLM, adding more Chinese to text-only pre-training | mLLaVA prompted in Chinese · VQA region | Western bias: 1.29 → 1.16 | Helped |
| Use Baichuan2 instead of Llama2 as the base LLM, adding more Chinese to text-only pre-training | mLLaVA prompted in Chinese · ArtELingo opposite valence | Western bias: 2.47 → 1.23 | Helped |
| Use Baichuan2 instead of Llama2 as the base LLM, adding more Chinese to text-only pre-training | mLLaVA prompted in Chinese · ArtELingo all | Western bias: 1.82 → 1.16 | Helped |
| Use Baichuan2 instead of Llama2 as the base LLM, adding more Chinese to text-only pre-training | mLLaVA prompted in English · ArtELingo opposite valence | Western bias: 3.45 → 2.71 | Helped |
| Use Baichuan2 instead of Llama2 as the base LLM, adding more Chinese to text-only pre-training | mLLaVA prompted in English · ArtELingo all | Western bias: 2.20 → 1.98 | Helped |
| Prompt in Chinese instead of English | mLLaVA with Baichuan2 base LLM · VQA en/zh | Western bias: 1.45 → 1.17 | Helped |
| Prompt in Chinese instead of English | mLLaVA with Baichuan2 base LLM · VQA region | Western bias: 1.27 → 1.16 | Helped |
| Prompt in Chinese instead of English | mLLaVA with Llama2 base LLM · VQA | Western bias | No clear effect |
| Prompt in Chinese instead of English | mLLaVA with Llama2 base LLM · ArtELingo opposite valence | Western bias: 3.45 → 2.47 | Helped |
| Prompt in Chinese instead of English | mLLaVA with Llama2 base LLM · ArtELingo all | Western bias: 2.20 → 1.82 | Helped |
| Prompt in Chinese instead of English | mLLaVA with Baichuan2 base LLM · ArtELingo opposite valence | Western bias: 2.72 → 1.23 | Helped |
Limitations
- Culture is proxied by country of image and annotator language, a simplification.
- Controlled language study limited to English and Chinese due to lack of comparable LLMs.
- Chinese fusion corpus is machine-translated.
Experimental setup
Datasets: Dollar Street A-OKVQA CVQA ArtELingo CC3M MS-COCO
Benchmarks: Dollar Street A-OKVQA CVQA ArtELingo
Metrics: accuracy bias ratio (Western / East Asian) F1
Baselines: BLIP-2 LLaVA-1.5 Qwen-VL mBLIP
Builds on: LLaVA LoRA logit lens
Closest papers by meaning
- No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- Can We Talk Models Into Seeing the World Differently? (ICLR 2025 · similarity 0.91)
- Semantic and Expressive Variations in Image Captions Across Languages (CVPR 2025 · similarity 0.91)
- Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.91)
- Words or Vision: Do Vision-Language Models Have Blind Faith in Text? (CVPR 2025 · similarity 0.90)
- Multilingual Diversity Improves Vision-Language Representations (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.