Research map / Vision-language and multimodal / Multimodal LLMs
LRM-LLaVA: Overcoming the Modality Gap of Multilingual Large Language-Vision Model for Low-Resource Languages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LRM-LLaVA, a 13B multilingual vision-language model for 10 languages that uses a projector plus cross-modal regularizer and three translated multilingual VQA task instructions, and builds four translated multilingual benchmarks.
Problem
Multilingual large vision-language models perform well on English but poorly on non-English, especially low-resource languages, due to a modality gap between visual inputs and multilingual text caused by scarce multilingual image-text training data.
Approach
Translates English VQA/instruction data into 10 languages and trains with three multilingual task instructions (monolingual VQA, English-pivot VQA, bilingual VQA) in a two-stage pipeline, with a cross-modal regularizer encouraging consistent representations across modalities to narrow the modality gap.
Key results
- On multilingual MMBench, LRM-LLaVA averages 66.1 over 9 non-English languages versus 62.1 for LVIS-Instruct4V-style baselines' best (ShareGPT4V 63.4-range average), with Czech 66.1 vs 61.0 for ShareGPT4V.
- On multilingual POPE, LRM-LLaVA scores 83-87 across languages, including Croatian 86.0 where LLaVA-v1.5 scores 44.8.
- Removing all task instructions reduces the non-English average on multilingual MMBench from 66.1 to 61.6 while English stays at about 70.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use all three multilingual task instructions instead of removing all instructions (ablation) | LRM-LLaVA with Vicuna-13B · Multilingual MMBench (9 non-English languages, Avg) | score: 61.6 -> 66.1 | Helped |
| Use all three multilingual task instructions instead of removing all instructions (ablation) | LRM-LLaVA with Vicuna-13B · Multilingual MMBench (English) | score: 70.2 -> 70.1 | No clear effect |
| Add MVQA instruction during pre-training (ablation) | LRM-LLaVA with Vicuna-13B · Multilingual MMBench (9 non-English languages, Avg) | score: 65.0 -> 66.1 | Helped |
| Add MVQA instruction during fine-tuning (ablation) | LRM-LLaVA with Vicuna-13B · Multilingual MMBench (9 non-English languages, Avg) | score: 63.8 -> 66.1 | Helped |
| Add EP-VQA instruction (ablation) | LRM-LLaVA with Vicuna-13B · Multilingual MMBench (9 non-English languages, Avg) | score: 63.9 -> 66.1 | Helped |
| Add BVQA instruction (ablation) | LRM-LLaVA with Vicuna-13B · Multilingual MMBench (9 non-English languages, Avg) | score: 65.2 -> 66.1 | Helped |
| Use multilingual training data and the proposed framework rather than English-only training (ablation) | Vicuna-13B LVLM · Multilingual MMBench (9 non-English languages, Avg) | score: 61.6 -> 66.1 | Helped |
| Use multilingual training data and the proposed framework rather than English-only training (ablation) | Vicuna-13B LVLM · Multilingual MMBench (English) | score: 70.2 -> 70.1 | No clear effect |
| Use multilingual training data and the proposed framework rather than English-only training (ablation) | Vicuna-7B LVLM · Multilingual MMBench (9 non-English languages, Avg) | score: 60.9 -> 64.7 | Helped |
| Use multilingual training data and the proposed framework rather than English-only training (ablation) | LLaMA2-13B LVLM · Multilingual MMBench (9 non-English languages, Avg) | score: 58.7 -> 62.1 | Helped |
| Use multilingual training data and the proposed framework rather than English-only training (ablation) | LLaMA3-8B LVLM · Multilingual MMBench (9 non-English languages, Avg) | score: 62.3 -> 66.0 | Helped |
| Increase non-English training-data mixture from 100% to 150% (ablation) | LRM-LLaVA · Multilingual MMBench (English) | score | Hurt |
Surprising findings
- Removing all task instructions leaves English performance unchanged while lowering non-English averages by 4.5 points.
- Mixing 150% non-English data reduced English performance significantly without further non-English gains.
Limitations
- Non-English training data is machine-translated (Google Translate), and math tasks were trained only on English, so translation quality is a bottleneck.
- Evaluation benchmarks are translated from English short-answer datasets, limiting coverage of open-ended generation.
- Cross-modal regularizer is described only qualitatively; its specific loss is not formalized beyond the task instructions.
Experimental setup
Datasets: ShareGPT4V-PT LLaVA-1.5 mixtures ChartQA GeoQA DocVQA DVQA AI2D MME MMBench POPE SEED-Bench
Benchmarks: Multilingual MME Multilingual POPE Multilingual SEED-Bench (Image) Multilingual MMBench
Metrics: accuracy/score BLEU
Baselines: InstructBLIP LLaVA-1.5 LVIS-Instruct4V ShareGPT4V Qwen-VL-Chat
Builds on: LLaVA-1.5 ShareGPT4V Vicuna
Closest papers by meaning
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.91)
- DiM$^3$: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging (NeurIPS 2026 · similarity 0.90)
- IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs (ICLR 2026 · similarity 0.90)
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (ICML 2024 · similarity 0.90)
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.89)
- Multilingual Diversity Improves Vision-Language Representations (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.