Research map / Vision-language and multimodal / Multimodal LLMs
All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ALM-bench, a 22.7K-sample multimodal VQA benchmark spanning 100 languages and 13 cultural domains (plus 6 generic), and evaluates 16 LMMs on it using GPT-4o as judge.
Problem
Existing LMM benchmarks cover few languages and cultures, are mostly English or high-resource, and often restricted to one question type, so they fail to measure cultural and multilingual visual understanding, especially for low-resource languages.
Approach
A culturally curated, human-verified multilingual VQA benchmark covering 100 languages from 73 countries with four question formats (MCQ, true/false, short and long VQA), built from web-sourced cultural images with GPT-4o-generated QAs that native speakers verify and translate.
Key results
- GPT-4o achieves 78.8% overall accuracy, versus 51.9% for the best open-source model GLM-4V-9B (about 27% gap).
- Open-source models drop sharply on low-resource languages, e.g., GLM-4V-9B falls from 80.3% on English to 15.6% on Amharic.
- Removing images drops GPT-4o performance by 27.3% absolute, showing the benchmark requires visual context.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add country-specific information to the prompt | GPT-4o · ALM-bench | accuracy: 80.96% -> 83.57% | Helped |
| Add country-specific information to the prompt | Gemini-1.5-Pro · ALM-bench | accuracy: 76.19% -> 81.52% | Helped |
| Add country-specific information to the prompt | GLM-4V-9B · ALM-bench | accuracy: 56.41% -> 56.78% | Helped |
| Add country-specific information to the prompt | Qwen2-VL · ALM-bench | accuracy: 52.57% -> 53.97% | Helped |
| Remove the image and provide only textual questions (ablation) | GPT-4o · ALM-bench, 50 sampled languages | performance: -27.3% | Hurt |
| Remove the image and provide only textual questions (ablation) | Qwen2-7B · ALM-bench, 50 sampled languages | performance: 13% absolute; 24.8% relative drop | Hurt |
Surprising findings
- Gemini-1.5-Pro does not deteriorate significantly on low-resource languages, unlike other models.
- Closed-source models perform better on long VQA than short VQA, while open-source models show the opposite trend.
Limitations
- Translations and QA generation rely on GPT-4o, which makes frequent semantic, grammatical and cultural errors in many languages.
- Cultural images are sourced from the web and may carry representation biases in underrepresented domains.
- GPT-4o as judge may bias scoring; consistency checked only on a 20-language sample with Llama-3.1-8B.
Experimental setup
Datasets: LLaVA-Bench (In-the-Wild)
Benchmarks: ALM-bench
Metrics: accuracy correctness consistency fluency relevance
Baselines: GPT-4o Gemini-1.5-Pro GLM-4V-9B Qwen2-VL LLaVA-OneVision Molmo InternVL2-8B Pixtral-12B
Builds on: LLaVA-Bench (In-the-Wild) CVQA World Values Survey
Closest papers by meaning
- Culture In a Frame: C$^3$B as a Comic-Based Benchmark for Multimodal Culturally Awareness (ICLR 2026 · similarity 0.90)
- Beyond Aesthetics: Cultural Competence in Text-to-Image Models (NeurIPS 2024 · similarity 0.89)
- Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation (ICLR 2026 · similarity 0.89)
- Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages (ICLR 2025 · similarity 0.89)
- IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs (ICLR 2026 · similarity 0.89)
- See It from My Perspective: How Language Affects Cultural Bias in Image Understanding (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.