Research map / Vision-language and multimodal / Visual question answering
CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds CVQA, a 10k-question culturally diverse multilingual multiple-choice VQA benchmark covering 30 countries and 31 languages, collected by native speakers, and benchmarks open and closed MLLMs on it.
Problem
Existing VQA benchmarks are mostly English-centric and Western-centric in images, lacking low-resource languages and cultural diversity in visual content.
Approach
Crowdsource culturally grounded images and multiple-choice questions written in local languages (with English translations) from native speakers and cultural experts across 39 country-language pairs, with peer validation, rather than translating questions over shared Western images.
Key results
- GPT-4o achieves the highest accuracy on CVQA with 75.4% (English prompts) and 74.3% (local-language prompts).
- LLaVA-1.5-7B, the best open model, reaches 49.6% with English prompts and 35.5% with local-language prompts, more than 10 points below closed models.
- Switching from multiple-choice to open-ended answer selection drops LLaVA-1.5-7B from 49.6% to about 30% average accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Ask questions in local languages instead of English | CVQA evaluation of open and closed MLLMs · CVQA | accuracy: LLaVA-1.5-7B: 49.6 -> 35.5; M-CLIP: 38.0 -> 33.7; CLIP: 42.7 -> 30.6; mBLIP-mT0: 31.3 -> 30.9; mBLIP-BLOOMZ: 39.3 -> 32. | Hurt |
| Use a location-aware prompt instead of a location-agnostic prompt (ablation) | LLaVA-1.5-7B, M-CLIP, CLIP, mBLIP-mT0, mBLIP-BLOOMZ, and InstructBLIP · CVQA | accuracy: LLaVA-1.5-7B: 48.3 -> 49.6 (EN), 34.7 -> 35.5 (LOC); M-CLIP: 38.1 -> 38.0 (EN), 33.8 -> 33.7 (LOC); CLIP: 43.8 -> 42.7 ( | No clear effect |
| Use web images instead of self-made images (ablation) | CVQA evaluation of LLaVA-1.5-7B and CLIP · CVQA image-source subsets | accuracy: LLaVA-1.5-7B EN: 48.8 -> 49.7; CLIP EN: 41.2 -> 43.1 | Helped |
| Use web images instead of self-made images (ablation) | CVQA evaluation across models · CVQA image-source subsets | accuracy: M-CLIP EN: 38.1 -> 37.4; mBLIP-mT0 EN: 31.2 -> 31.9; mBLIP-BLOOMZ EN: 40.1 -> 38.7; InstructBLIP EN: 48.3 -> 49.1 | Mixed |
| Remove multiple-choice options from the prompt (ablation) | LLaVA-1.5-7B · CVQA | accuracy: 49.6% to 30% | Hurt |
Surprising findings
- Location-aware prompts give only marginal gains over location-agnostic prompts, suggesting location information has no significant effect on MLLMs.
- Models often score higher on web images than on self-made images (e.g., LLaVA-1.5-7B 49.7% vs 48.8% in English).
Limitations
- Covers only a fraction of the world's languages and regions; no English-centric baseline.
- Data scale (about 10k questions) is too small for training, so the benchmark is evaluation-only.
- Country-level granularity and a bounded definition of culture as common-ground knowledge.
Experimental setup
Datasets: CVQA
Benchmarks: CVQA VQAv2 GQA VizWiz SciQA-IMG TextVQA
Metrics: accuracy
Baselines: LLaVA-1.5-7B CLIP M-CLIP mBLIP InstructBLIP Gemini-1.5-Flash GPT-4o
Builds on: OK-VQA Masakhane NusaCrowd AI4Bharat
Closest papers by meaning
- BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining (AAAI 2024 · similarity 0.92)
- IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs (ICLR 2026 · similarity 0.92)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.91)
- Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (NeurIPS 2024 · similarity 0.90)
- FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.90)
- Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.