Research map / Vision-language and multimodal / Multimodal LLMs

Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Kaleidoscope, a 20,911-question, 18-language in-language multiple-choice exam benchmark with images, and evaluates 13 open and closed VLMs, finding weak performance on low-resource languages, STEM, and non-Latin scripts.

Problem

Vision-language model evaluation is dominated by English and translated benchmarks, which miss cultural nuance and multilingual multimodal coverage.

Approach

Collects real-world, original-language exam questions (55% requiring images) via an open science community across 18 languages and 14 subjects, avoiding translation, and analyzes performance by modality, image type, subject, language, and script.

Key results

• Claude 3.5 Sonnet achieves the highest overall accuracy at 62.91%, with Gemini 1.5 Pro at 62.10% and GPT-4o at 58.32%; best open model Qwen2.5-VL-72B reaches 52.94%.
• Multimodal accuracy is lower than text-only for all models; GPT-4o drops 21.6 points from text-only to multimodal.
• Humanities & Social Sciences accuracy averages 83.7% versus 59.2% for STEM (best scores across models).

Surprising findings

• Qwen2.5-VL-7B with a random image drops only modestly and with no image still scores 33.4%, well above the 25% chance baseline, suggesting reliance on text cues.
• Adding synthetic captions and OCR text does not uniformly help; it hurts Qwen2.5-VL-7B on Table (-6.3 points) while helping on Text and Formula.

Limitations

• Language coverage is imbalanced and limited; many low-resource languages are missing due to scarce exam sources.
• Difficulty levels and grade levels are not uniformly controlled across exams and languages.
• MCQA format has issues such as answer-choice bias exploitation and order sensitivity.

Experimental setup

Datasets: Kaleidoscope

Benchmarks: Kaleidoscope

Metrics: accuracy valid-response accuracy format error rate

Baselines: Claude 3.5 Sonnet Gemini 1.5 Pro GPT-4o Qwen2.5-VL Aya-Vision Molmo-7B-D Pangea-7B

Builds on: CVQA M3Exam EXAMS-V MMLU-ProX

Closest papers by meaning

• VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.90)
• IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs (ICLR 2026 · similarity 0.90)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.89)
• Multilingual Diversity Improves Vision-Language Representations (NeurIPS 2024 · similarity 0.89)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.89)
• All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages (CVPR 2025 · similarity 0.89)