Research map / Vision-language and multimodal / Multimodal LLMs
Parrot: Multilingual Visual Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Parrot, a LLaVA-style MLLM that uses text-guided cross-attention and a language-expert MoE to convert English-biased visual tokens into language-specific ones, plus the MMMB six-language multimodal benchmark.
Problem
Multimodal LLMs lose non-English ability after English-centric visual instruction tuning ("multilingual erosion"), partly because visual tokens are misaligned with non-English text tokens.
Approach
Condition visual tokens on text embeddings via cross-attention, then route them through a language-expert MoE (one expert per language) and add the result residually, so that only ~10K non-English image-text samples per language are needed for multilingual alignment.
Key results
- Parrot-Qwen2-7B reaches the best or second-best accuracy on MMMB and MMBench across six languages, e.g. 76.55 on Arabic MMMB versus 74.07 for Qwen2-VL.
- Parrot-Qwen1.5-7B improves over LLaVA-1.5 with the same data, e.g. MMMB Arabic 62.69 vs 43.50 and Turkish 58.01 vs 46.43 (as reported in the table).
- Using ~2M total training samples, Parrot is reported to be competitive on LLaVA-Bench in English and Chinese, with the paper citing use of under 1% of the training data of compared multilingual models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Parrot instead of Qwen2-VL | Qwen2-7B MLLM · MMMB | accuracy | Mixed |
| use Parrot instead of Qwen2-VL | Qwen2-7B MLLM · MMBench | accuracy | Mixed |
| add multilingual data to LLaVA (ablation) | LLaVA · MMMB | accuracy: 67.1&58.8&59.8&43.5&46.4&59.1 -> 67.0&59.1&60.3&44.2&48.1&59.7 | Mixed |
| add Chinese training data (ablation) | LLaVA-1.5-finetune · MMBench | accuracy: 69.4 -> 69.2; 66.6 -> 68.6; 60.3 -> 64.1; 55.3 -> 59.1; 52.1 -> 50.9; 60.7 -> 61.6 | Mixed |
| add Portuguese training data along with Chinese (ablation) | LLaVA-1.5-finetune · MMBench | accuracy: +1.7; +3.8; +5.1; +2.6; +0.0; +2.2 | Helped |
| add Arabic training data along with Chinese and Portuguese (ablation) | LLaVA-1.5-finetune · MMBench | accuracy: +1.6; +2.0; +5.4; +3.3; +0.1; +1.5 | Helped |
| add Turkish training data along with Chinese, Portuguese, and Arabic (ablation) | LLaVA-1.5-finetune · MMBench | accuracy: +1.0; +2.1; +4.6; +5.9; +7.6; +1.3 | Helped |
| add Russian training data along with Chinese, Portuguese, Arabic, and Turkish (ablation) | LLaVA-1.5-finetune · MMBench | accuracy: +1.3; +3.8; +4.8; +2.5; +6.3; +3.3 | Helped |
| increase multilingual training data from 10K to 70K per language (ablation) | Parrot-7B · MMMB | accuracy: 70.0 -> 70.3; 68.1 -> 68.4; 67.3 -> 68.3; 62.7 -> 65.7; 58.0 -> 63.2; 66.3 -> 67.4 | Mixed |
| scale the LLM backbone from 7B to 32B (ablation) | Parrot · MMMB | accuracy: 70.0 -> 76.3; 68.1 -> 75.4; 67.3 -> 73.8; 62.7 -> 72.1; 58.0 -> 71.2; 66.3 -> 73.5 | Helped |
| use Chinese-CLIP instead of OpenAI-CLIP (ablation) | Parrot-Qwen1.5-7B · MMMB and MMBench | accuracy | Mixed |
| use the MoE module for language-specific visual-token alignment (ablation) | Parrot · MMBench | accuracy | Helped |
Surprising findings
- Adding the same multilingual data to LLaVA gives only marginal gains (e.g. Arabic 43.5 to 44.2), while Parrot gains substantially, suggesting data alone is not sufficient.
- A naive translate-then-answer baseline causes seesaw effects, degrading Russian and Portuguese while improving Chinese.
Limitations
- Benchmark translations and questions rely on GPT-4 translation plus human calibration, and the paper acknowledges possible residual noise.
- Evaluation is limited to six languages and the 32B scaling experiment is only on MMMB.
- Claims of reduced data use rely on approximate counts and the 'less than 1%' comparison is not fully systematic.
Experimental setup
Datasets: LLaVA-1.5-pretrain Laion-Caption CC12M-Caption LLaVA-1.5-finetune ShareGPT4V-zh ShareGPT4V-pt ShareGPT4V-ar ShareGPT4V-tr ShareGPT4V-ru MMBench ScienceQA MME
Benchmarks: MMMB MMBench MME MMStar ScienceQA RealWorldQA SEED-Bench LLaVA-Bench
Metrics: accuracy (circular yes/no) GPT-4 judged score
Baselines: LLaVA-1.5 LLaVA-NeXT Qwen2-VL LLaVA-OneVision Idefics3-8B DeepSeek-VL-7B GLM-4v-9B Qwen-VL-Chat mPLUG-Owl2 Monkey ShareGPT4V InstructBLIP
Builds on: LLaVA ShareGPT4V MMBench Qwen2
Closest papers by meaning
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.89)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.89)
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (ICML 2024 · similarity 0.89)
- ROSE: Rotate Your Large Language Model to See (CVPR 2026 · similarity 0.89)
- Visual Perception by Large Language Model’s Weights (NeurIPS 2024 · similarity 0.88)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.