Research map / Vision-language and multimodal / Multimodal LLMs

Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Pangea, a 7B multilingual multimodal LLM trained on PangeaIns (6M instructions in 39 languages, including culturally grounded LAION-derived data) and evaluated on PangeaBench (14 datasets, 47 languages).

Problem

Multimodal LLMs are dominated by English- and western-centric data and evaluation, leaving most world languages and cultural contexts underrepresented, with few open multilingual MLLMs and weak cultural evaluation.

Approach

Combine machine-translated English multimodal instructions (translated with Gemini 1.5 Pro) with a new pipeline that filters culturally informative LAION-Multi images via LLM scoring, recaptions them, and generates multilingual cross-cultural instructions; pair this with a fine-grained rubric-based multilingual chat benchmark (xChatBench).

Key results

• Pangea-7B averages 59.9 on English and 52.8 on multilingual across PangeaBench multimodal tasks, +0.4 (en) and +10.9 (multilingual) over the best open model.
• On xChatBench multilingual, Pangea-7B scores 35.8 versus 28.5 for the best open baseline (Llava-OV-7B) on the multilingual subset.
• On text-only multilingual benchmarks, Pangea-7B averages 54.3 multilingual versus 54.5 for its backbone Qwen2-7B-Instruct, with large MGSM gains (47.4 vs 40.4 multilingual).

Measured effects reported in the paper

Surprising findings

• Using 100% English data in training sharply reduces multilingual performance, with multilingual gains peaking near 40% English data.
• Many English-centric models respond in English regardless of query language, so the authors penalize language mismatch by assigning score 1.

Limitations

• Machine translation (Gemini 1.5 Pro) may miss nuanced cultural subtleties and struggles with low-resource languages.
• Gaps remain versus GPT-4o and Gemini-1.5-Pro in multimodal chat and complex reasoning.
• Multilingual OCR and low-resource language performance remain challenging.

Experimental setup

Datasets: PangeaIns LLaVA LCS-558K LAION-Multi Chinese ALLaVA-4V MTVQA LLaVA-Japanese-Instruct STAIR Captions Russian GQA French Doc-VQA

Benchmarks: PangeaBench xChatBench M-LlavaBench XM100 CVQA MaRVL xGQA MaXM xMMMU M3Exam FLORES-Sub TyDiQA

Metrics: accuracy LLM-judge score (1-5 normalized to 0-100) CIDEr BLEU

Baselines: Llava-1.5-7B Llava-Next-7B Phi-3.5-Vision Cambrian-8B Llava-OV-7B Molmo-7B-D Llama3.2-11B PaliGemma-3B PALO-7B mBLIP BLOOMZ Gemini-1.5-Pro GPT4o

Builds on: LLaVA-Next LLaVA Qwen2-7B-Instruct LAION-Multi MMMU

Closest papers by meaning

• MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.90)
• Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.90)
• XIFBench: Evaluating Large Language Models on Multilingual Instruction Following (NeurIPS 2025 · similarity 0.89)
• DiM$^3$: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging (NeurIPS 2026 · similarity 0.89)
• All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages (CVPR 2025 · similarity 0.89)
• CultureLLM: Incorporating Cultural Differences into Large Language Models (NeurIPS 2024 · similarity 0.88)